You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/05/27 08:24:44 UTC
[PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
englefly opened a new pull request, #63736:
URL: https://github.com/apache/doris/pull/63736
…sted column pruning
Core changes:
- LazyMaterializeTopN: find projects below TopN with PreferPushDownProject or variant subColPath expressions, simplify them to expose base columns as lazy candidates. Pull up expressions above Materialize for lazy mat.
- LazySlotPruning: extract shouldPruneChild() for safe override when logical properties are stale after plan restructuring
- OperativeColumnDerive: skip PreferPushDownProject input slots from operative propagation so struct/variant columns can be lazy
- PhysicalLazyMaterialize: propagate access paths to lazy output slots
- MaterializationNode: use materializeTupleDescriptor for nested column info display (was outputTupleDesc, never set)
- PlanNode: add subColLables to printNestedColumns for variant
Tests cover struct (struct_element + pruned type) and variant (element_at + sub path) lazy materialization with nested column pruning.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4633454765
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4627625072
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4563335775
<details>
<summary>TPC-DS: <b>Total hot run time: 172620 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 80a6ff0127dca6e6fe6f5ed9f4e0477ab0281d2c, data reload: false
query5 4336 685 531 531
query6 365 240 202 202
query7 4306 577 318 318
query8 331 239 223 223
query9 8773 4109 4072 4072
query10 471 361 300 300
query11 5820 2436 2243 2243
query12 199 126 126 126
query13 1308 598 446 446
query14 6169 5540 5209 5209
query14_1 4558 4531 4504 4504
query15 219 209 183 183
query16 991 463 431 431
query17 1168 749 618 618
query18 2529 483 360 360
query19 221 219 171 171
query20 144 144 132 132
query21 219 146 127 127
query22 13703 13518 13374 13374
query23 17306 16436 16220 16220
query23_1 16380 16310 16412 16310
query24 7549 1811 1390 1390
query24_1 1364 1366 1314 1314
query25 587 496 461 461
query26 1388 357 186 186
query27 2732 613 355 355
query28 4450 2011 2017 2011
query29 996 640 540 540
query30 307 245 205 205
query31 1139 1090 959 959
query32 89 77 81 77
query33 573 374 314 314
query34 1188 1179 677 677
query35 817 806 769 769
query36 1400 1409 1225 1225
query37 158 111 92 92
query38 3233 3173 3051 3051
query39 932 916 889 889
query39_1 874 880 900 880
query40 243 149 126 126
query41 65 64 63 63
query42 110 111 112 111
query43 344 349 300 300
query44
query45 216 203 195 195
query46 1086 1224 741 741
query47 2344 2350 2236 2236
query48 422 433 294 294
query49 626 496 391 391
query50 1009 347 247 247
query51 4389 4265 4457 4265
query52 108 108 95 95
query53 255 279 206 206
query54 315 287 254 254
query55 94 92 90 90
query56 321 300 314 300
query57 1432 1403 1345 1345
query58 303 274 277 274
query59 1666 1711 1486 1486
query60 321 328 326 326
query61 164 159 158 158
query62 698 647 584 584
query63 260 204 206 204
query64 2386 813 639 639
query65
query66 1729 473 359 359
query67 29821 29709 29555 29555
query68
query69 503 350 304 304
query70 1036 1010 1028 1010
query71 322 318 272 272
query72 3050 2692 2443 2443
query73 814 812 441 441
query74 5137 4994 4804 4804
query75 2839 2593 2260 2260
query76 2293 1171 789 789
query77 410 421 340 340
query78 12464 12388 11943 11943
query79 1464 1095 760 760
query80 662 549 464 464
query81 454 279 250 250
query82 1352 160 130 130
query83 365 281 259 259
query84 262 146 109 109
query85 888 534 465 465
query86 416 363 326 326
query87 3448 3377 3259 3259
query88 3678 2757 2685 2685
query89 453 393 345 345
query90 1958 192 185 185
query91 180 170 137 137
query92 84 82 75 75
query93 1479 1453 874 874
query94 553 352 317 317
query95 700 489 349 349
query96 1125 748 345 345
query97 2722 2730 2606 2606
query98 238 232 224 224
query99 1190 1153 1014 1014
Total cold run time: 255396 ms
Total hot run time: 172620 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4565452521
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4642424345
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4592636526
<details>
<summary>TPC-H: <b>Total hot run time: 29103 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cefd39843930a5392f14a48b3c47d033bb40e59d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17705 3987 3990 3987
q2 q3 10751 1352 832 832
q4 4684 463 349 349
q5 7537 842 596 596
q6 185 166 138 138
q7 808 852 640 640
q8 9834 1614 1615 1614
q9 7032 4524 4461 4461
q10 6737 1814 1507 1507
q11 437 271 250 250
q12 639 427 288 288
q13 18149 3366 2778 2778
q14 265 256 240 240
q15 q16 821 782 704 704
q17 946 902 950 902
q18 6738 5814 5743 5743
q19 1325 1266 1113 1113
q20 502 402 262 262
q21 5834 2596 2404 2404
q22 438 352 295 295
Total cold run time: 101367 ms
Total hot run time: 29103 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4343 4275 4379 4275
q2 q3 4486 4976 4372 4372
q4 2074 2214 1388 1388
q5 4418 4284 4302 4284
q6 234 174 129 129
q7 1762 1930 1739 1739
q8 2537 2144 2075 2075
q9 7978 7865 8027 7865
q10 4809 4742 4311 4311
q11 634 416 395 395
q12 739 746 554 554
q13 3558 3521 3042 3042
q14 319 313 284 284
q15 q16 756 730 666 666
q17 1304 1330 1339 1330
q18 7983 7329 7109 7109
q19 1117 1186 1102 1102
q20 2211 2218 1942 1942
q21 5254 4572 4517 4517
q22 518 455 418 418
Total cold run time: 57034 ms
Total hot run time: 51797 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4554395326
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4554441882
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4566453019
# FE UT Coverage Report
Increment line coverage ` 64.68% (174/269)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_835b604c4ad63b88f63c6caeb098d12c98a4be08/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_835b604c4ad63b88f63c6caeb098d12c98a4be08/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4624606062
# FE UT Coverage Report
Increment line coverage ` 67.88% (336/495)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_9c005743605d8bb37f81400c7996fe80d09a8875/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_9c005743605d8bb37f81400c7996fe80d09a8875/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4631950743
<details>
<summary>TPC-DS: <b>Total hot run time: 168528 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90b0103ac670a6a7745e940394f4a283481be422, data reload: false
query5 4322 633 488 488
query6 436 193 193 193
query7 4854 555 315 315
query8 378 216 214 214
query9 8752 3984 3975 3975
query10 465 334 260 260
query11 5959 2371 2181 2181
query12 157 107 102 102
query13 1264 633 465 465
query14 6410 5418 5052 5052
query14_1 4396 4377 4408 4377
query15 215 203 180 180
query16 1022 472 454 454
query17 1135 732 621 621
query18 2737 490 353 353
query19 211 189 153 153
query20 113 118 113 113
query21 220 143 124 124
query22 13543 13720 13355 13355
query23 17365 16521 16214 16214
query23_1 16313 16234 16344 16234
query24 7640 1762 1294 1294
query24_1 1322 1278 1286 1278
query25 544 438 386 386
query26 1302 308 166 166
query27 2672 553 343 343
query28 4420 1996 1990 1990
query29 1056 586 488 488
query30 309 239 207 207
query31 1113 1084 950 950
query32 112 65 64 64
query33 532 317 248 248
query34 1185 1152 639 639
query35 752 857 681 681
query36 1405 1416 1225 1225
query37 150 105 95 95
query38 3232 3142 3046 3046
query39 918 924 921 921
query39_1 905 865 859 859
query40 218 122 100 100
query41 63 60 62 60
query42 93 93 92 92
query43 314 323 272 272
query44
query45 196 182 180 180
query46 1107 1245 740 740
query47 2350 2384 2266 2266
query48 396 410 300 300
query49 630 473 376 376
query50 981 379 254 254
query51 4391 4404 4229 4229
query52 87 92 76 76
query53 239 262 191 191
query54 264 249 206 206
query55 77 80 75 75
query56 239 216 211 211
query57 1413 1407 1316 1316
query58 237 216 210 210
query59 1544 1646 1437 1437
query60 291 242 230 230
query61 154 155 159 155
query62 712 648 569 569
query63 222 187 184 184
query64 2507 795 623 623
query65
query66 1740 461 343 343
query67 30008 29647 29452 29452
query68
query69 429 303 260 260
query70 1001 961 954 954
query71 305 229 195 195
query72 3008 2668 2331 2331
query73 825 735 443 443
query74 5168 4900 4754 4754
query75 2664 2595 2266 2266
query76 2324 1152 815 815
query77 351 369 273 273
query78 12363 12216 11930 11930
query79 1277 1072 760 760
query80 526 459 388 388
query81 450 287 246 246
query82 238 160 124 124
query83 376 273 247 247
query84
query85 852 549 440 440
query86 334 299 294 294
query87 3357 3341 3171 3171
query88 3629 2726 2681 2681
query89 402 381 335 335
query90 2188 181 175 175
query91 176 161 133 133
query92 65 61 55 55
query93 1429 1386 916 916
query94 522 352 306 306
query95 674 404 337 337
query96 1007 764 356 356
query97 2692 2696 2567 2567
query98 216 214 211 211
query99 1152 1207 1033 1033
Total cold run time: 250723 ms
Total hot run time: 168528 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3351247484
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,642 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+ final Map<ExprId, NamedExpression> passThroughExprByDeduplicatedExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+
+ void addPassThroughExprForDeduplicatedExpr(NamedExpression expr) {
+ passThroughExprByDeduplicatedExpr.put(expr.getExprId(), expr);
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ final Map<Slot, Expression> pullUpExprReplaceMap = new LinkedHashMap<>();
+ /**
+ * When collectFromNode encounters a nested TopN, it saves the current
+ * blockedExprIds (accumulated from outer nodes) so that visitLogicalTopN
+ * for the inner TopN can merge them into its fresh blocked set.
+ */
+ final Map<LogicalTopN, Set<ExprId>> outerBlockedByTopN = new IdentityHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ void addPullUpExprReplace(NamedExpression expr) {
+ if (expr instanceof Alias) {
+ pullUpExprReplaceMap.putIfAbsent(expr.toSlot(), expr.child(0));
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ // If this is a nested TopN, merge in the outer blocked set that was
+ // saved by collectFromNode when it encountered this TopN. This
+ // ensures that slots consumed by outer operators (e.g. join
+ // conditions above this TopN) also block pull-up from projects
+ // under this TopN.
+ Set<ExprId> outerBlocked = context.outerBlockedByTopN.remove(topN);
+ if (outerBlocked != null) {
+ blockedExprIds.addAll(outerBlocked);
+ }
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds, context);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ for (NamedExpression expr : info.allPulledUpExprs) {
+ context.addPullUpExprReplace(expr);
+ }
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds,
+ CollectorContext context) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds, context);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // Save the current blockedExprIds (accumulated from outer nodes
+ // such as outer TopN + intermediate Joins) so that the inner
+ // TopN's own visitLogicalTopN can merge them into its fresh
+ // blocked set. Without this, outer join condition slots would
+ // not block pull-up from projects under the inner TopN.
+ context.outerBlockedByTopN.put(inner, new HashSet<>(blockedExprIds));
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked, context);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked, context);
Review Comment:
This traversal still lets the current TopN pull expressions from the nullable side of an outer join, which is not semantics-preserving for expressions that are not null-propagating. For example, `SELECT x FROM l LEFT JOIN (SELECT id, coalesce(a, 0) AS x FROM r) rr ON l.id = rr.id ORDER BY l.id LIMIT 10`: for unmatched left rows, the original plan returns `NULL` for `x` because the right-side row is null-extended after the lower project runs. After this rule simplifies the right project to pass `a` and restores `coalesce(a, 0)` above the TopN/join, the same unmatched row evaluates `coalesce(NULL, 0)` and returns `0`. Inner-join tests do not cover this. Please treat outer joins as boundaries for expressions from their nullable child, or only allow pull-up there when the expression is proven safe under null extension, and add a negative test for a nullable-side `coalesce`/`ifnull` expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4612008448
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3317497710
##########
regression-test/suites/nereids_rules_p0/column_pruning/topn_lazy_nested_column_pruning.groovy:
##########
@@ -0,0 +1,342 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("topn_lazy_nested_column_pruning") {
+ sql """ set topn_lazy_materialization_threshold=1024; """
+ sql """ DROP TABLE IF EXISTS ncp_tbl """
+ sql """
+ CREATE TABLE ncp_tbl (
+ id INT,
+ str_col STRING NULL,
+ struct_col STRUCT<city: STRING, zip: INT> NULL,
+ arr_col ARRAY<INT> NULL,
+ map_col MAP<STRING, INT> NULL,
+ int_col INT NULL
+ ) ENGINE = OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+
+ sql """
+ INSERT INTO ncp_tbl VALUES
+ (1, 'hello', named_struct('city', null, 'zip', 10001), [1, 2, 3], {'a': 1, 'b': 2 }, 1)
+ """
+
+ sql """
+ drop table if exists vt;
+ CREATE TABLE IF NOT EXISTS vt (
+ id BIGINT NOT NULL,
+ s varchar(100) null,
+ payload VARIANT<
+ 'name' : STRING,
+ 'age' : INT
+ > NULL
+ )
+ ENGINE = OLAP
+ DUPLICATE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1",
+ "storage_format" = "V3"
+ );
+
+
+ INSERT INTO vt VALUES
+ (1, 'aaa', '{"name": "张三", "age": 25}'),
+ (2, 'bbb', '{"name": "李四", "age": 30}'),
+ (3, 'ccc', '{"name": "王五", "age": 28, "city": "北京"}');
+ """
+
+ // =============================================
+ // Test 1: STRUCT type - lazy mat + nested column pruning
+ // =============================================
+ explain {
+ sql """
+ select id, substring(struct_element(struct_col, 'city'), 1) as city
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ // struct_col lazy, scan only outputs id + rowId
+ contains("final projections: id[#0], __DORIS_GLOBAL_ROWID_COL__ncp_tbl[#6]")
+ // nested column pruning: struct_col pruned to city only
+ contains("nested columns:")
+ contains("pruned type: struct<city:text>")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+
+ // =============================================
+ // Test 2: STRUCT with select * - struct_col explicit in output
+ // =============================================
+ explain {
+ sql """
+ select *, substring(struct_element(struct_col, 'city'), 1) as city
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+
+ // =============================================
+ // Test 3: VARIANT type - lazy mat + sub path pruning
+ // =============================================
+ explain {
+ sql """
+ select id, s, substring(element_at(payload, 'name'), 1) as name
+ from vt
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ // payload lazy, scan only outputs id + rowId
+ contains("final projections: id[#0], __DORIS_GLOBAL_ROWID_COL__vt[#4]")
+ // sub path pruning for variant
+ contains("nested columns:")
+ contains("sub path: [name]")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__vt]")
+ }
+
+ // =============================================
+ // Test 4: STRUCT - verify actual query results
+ // =============================================
+ qt_struct_result """
+ select id, substring(struct_element(struct_col, 'city'), 1) as city
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+
+ // =============================================
+ // Test 5: VARIANT - verify actual query results
+ // =============================================
+ qt_variant_result """
+ select id, s, substring(element_at(payload, 'name'), 1) as name
+ from vt
+ order by id
+ limit 3
+ """
+
+ // =============================================
+ // Test 6: MAP subscript - lazy mat + nested column pruning
+ // =============================================
+ explain {
+ sql """
+ select id, element_at(map_col, 'a') as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ // map_col lazy, scan only outputs id + rowId
+ contains("final projections: id[#0], __DORIS_GLOBAL_ROWID_COL__ncp_tbl[#6]")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+
+ // =============================================
+ // Test 7: MAP subscript - verify actual query results
+ // =============================================
+ qt_map_result """
+ select id, element_at(map_col, 'a') as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+
+ // =============================================
+ // Test 8: ARRAY subscript - lazy mat + nested column pruning
+ // =============================================
+ explain {
+ sql """
+ select id, element_at(arr_col, 1) as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ // arr_col lazy, scan only outputs id + rowId
+ contains("final projections: id[#0], __DORIS_GLOBAL_ROWID_COL__ncp_tbl[#6]")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+
+ // =============================================
+ // Test 9: ARRAY subscript - verify actual query results
+ // =============================================
+ qt_array_result """
+ select id, element_at(arr_col, 1) as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+
+ // =============================================
+ // Test 10: Multi-level VARIANT nested - insert nested data
+ // =============================================
+ sql """
+ INSERT INTO vt VALUES
+ (4, 'ddd', '{"address": {"city": "上海", "zip": "200000"}}'),
+ (5, 'eee', '{"address": {"city": "北京", "zip": "100000"}}')
+ """
+
+ // =============================================
+ // Test 11: Multi-level VARIANT nested - explain
+ // Access payload['address']['city'] via two levels:
+ // inner payload['address'] → variant with subColPath [address]
+ // outer element_at(..., 'city') → final value
+ // =============================================
+ explain {
+ sql """
+ select id, element_at(payload['address'], 'city') as city
+ from vt
+ where id >= 4
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ // payload lazy, scan only outputs id + rowId
+ contains("final projections: id[#0], __DORIS_GLOBAL_ROWID_COL__vt[#4]")
+ // sub path pruning for variant: only read address sub-path during materialization
+ contains("nested columns:")
+ contains("sub path: [address.city]")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__vt]")
+ }
+
+ // =============================================
+ // Test 12: Multi-level VARIANT nested - verify query results
+ // =============================================
+ qt_variant_nested_result """
+ select id, element_at(payload['address'], 'city') as city
+ from vt
+ where id >= 4
+ order by id
+ limit 3
+ """
+
+ // =============================================
+ // Test 13: using_index=true with MAP subscript
+ // Verify that map/array lazy mat still works when
+ // topn_lazy_materialization_using_index is enabled.
+ // Regression test for the risk that MaterializeProbeVisitor
+ // .visitPhysicalProject skips alias→child slot tracing
+ // when using_index=true, which could prevent base columns
+ // from being probed as lazy candidates.
+ // =============================================
+ sql """ set topn_lazy_materialization_using_index = true; """
+ explain {
+ sql """
+ select id, element_at(map_col, 'a') as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+ qt_map_using_index_result """
+ select id, element_at(map_col, 'a') as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ sql """ set topn_lazy_materialization_using_index = false; """
+
+ // =============================================
+ // Test 14: using_index=true with ARRAY subscript
+ // =============================================
+ sql """ set topn_lazy_materialization_using_index = true; """
+ explain {
+ sql """
+ select id, element_at(arr_col, 1) as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+ qt_array_using_index_result """
+ select id, element_at(arr_col, 1) as val
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ sql """ set topn_lazy_materialization_using_index = false; """
+
+ // =============================================
+ // Test 15: STRUCT nested expr BEFORE id — verify column order preserved
+ // SELECT city_expr, id should produce [city, id] not [id, city]
+ // =============================================
+ explain {
+ sql """
+ select substring(struct_element(struct_col, 'city'), 1) as city, id
+ from ncp_tbl
+ order by id
+ limit 3
+ """
+ contains("VMaterializeNode")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__ncp_tbl]")
+ }
+ qt_struct_col_order_result """
Review Comment:
These three new `qt_` blocks (`qt_struct_col_order_result`, `qt_variant_col_order_result`, and `qt_map_col_order_result`) do not have corresponding sections in `regression-test/data/nereids_rules_p0/column_pruning/topn_lazy_nested_column_pruning.out`. The regression runner compares every `qt_` label against the `.out` file, so this suite will fail before it can validate the intended column-order behavior. Please regenerate/update the `.out` file to include `-- !struct_col_order_result --`, `-- !variant_col_order_result --`, and `-- !map_col_order_result --` with the actual ordered results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4566965344
# FE Regression Coverage Report
Increment line coverage ` 83.27% (224/269)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_835b604c4ad63b88f63c6caeb098d12c98a4be08_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_835b604c4ad63b88f63c6caeb098d12c98a4be08_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3341560961
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,624 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+ final Map<ExprId, NamedExpression> passThroughExprByDeduplicatedExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+
+ void addPassThroughExprForDeduplicatedExpr(NamedExpression expr) {
+ passThroughExprByDeduplicatedExpr.put(expr.getExprId(), expr);
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ final Map<Slot, Expression> pullUpExprReplaceMap = new LinkedHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ void addPullUpExprReplace(NamedExpression expr) {
+ if (expr instanceof Alias) {
+ pullUpExprReplaceMap.putIfAbsent(expr.toSlot(), expr.child(0));
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ for (NamedExpression expr : info.allPulledUpExprs) {
+ context.addPullUpExprReplace(expr);
+ }
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked);
+ }
+ }
+
+ // =========================================================================
+ // Pull-up eligibility
+ // =========================================================================
+
+ /**
+ * Check if a named expression can be pulled up above TopN.
+ * Eligible: Alias with non-trivial child, not blocked, no NoneMovableFunction.
+ */
+ static boolean canPullUp(NamedExpression ne) {
+ if (!(ne instanceof Alias)) {
+ return false;
+ }
+ Expression child = ((Alias) ne).child();
+ if (child instanceof Slot || child instanceof Literal) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof NoneMovableFunction)) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof Score)) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof SlotReference && ((SlotReference) e).getDataType().isVariantType())) {
+ return false;
+ }
+ return true;
+ }
+
+ private static boolean isBlockingNode(Plan node) {
+ return node instanceof LogicalAggregate
+ || node instanceof LogicalWindow
+ || node instanceof LogicalRepeat;
+ }
+
+ private static Set<ExprId> buildOrderKeyExprIds(LogicalTopN<?> topN) {
+ Set<ExprId> orderKeyExprIds = new HashSet<>();
+ for (OrderKey orderKey : topN.getOrderKeys()) {
+ Expression keyExpr = orderKey.getExpr();
+ orderKeyExprIds.addAll(keyExpr.getInputSlotExprIds());
+ if (keyExpr instanceof NamedExpression) {
+ orderKeyExprIds.add(((NamedExpression) keyExpr).getExprId());
+ }
+ }
+ return orderKeyExprIds;
+ }
+
+ /**
+ * Deduplicate pull-up expressions so that each expression in a Project is only
+ * pulled up to the outermost TopN that collects it.
+ *
+ * <p>Since {@link CollectorContext#topNToPullUpInfo} is a {@link LinkedHashMap}
+ * and the Collector visits the plan top-down, iteration order is outer-to-inner.
+ * We keep the first occurrence of each (project-reference, exprId) pair and
+ * remove duplicates from inner TopNs.
+ */
+ private static void deduplicatePullUps(CollectorContext context) {
+ // Use IdentityHashMap because we need to distinguish Project nodes by object
+ // reference, not by content equality.
+ Map<LogicalProject<? extends Plan>, Set<ExprId>> handled = new IdentityHashMap<>();
+
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> toRemove = new ArrayList<>();
+ for (Map.Entry<LogicalProject<? extends Plan>, List<NamedExpression>> entry
+ : info.projectToPulledUpExprs.entrySet()) {
+ LogicalProject<? extends Plan> project = entry.getKey();
+ Set<ExprId> projectHandled = handled.computeIfAbsent(project, k -> new HashSet<>());
+ for (NamedExpression expr : entry.getValue()) {
+ if (projectHandled.contains(expr.getExprId())) {
+ toRemove.add(expr);
+ } else {
+ projectHandled.add(expr.getExprId());
+ }
+ }
+ }
+ for (NamedExpression expr : toRemove) {
+ info.addPassThroughExprForDeduplicatedExpr(expr);
+ info.allPulledUpExprs.remove(expr);
+ for (List<NamedExpression> list : info.projectToPulledUpExprs.values()) {
+ list.removeIf(e -> e == expr);
+ }
+ info.projectToPulledUpExprs.entrySet().removeIf(e -> e.getValue().isEmpty());
+ info.baseSlotsByExpr.remove(expr.getExprId());
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 2: Replacer (bottom-up)
+ // =========================================================================
+
+ static class Replacer extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, CollectorContext context) {
+ LogicalProject<? extends Plan> rewritten = (LogicalProject<? extends Plan>) visit(project, context);
+
+ // Collect ALL pulled-up expressions across ALL PullUpInfos for this
+ // project. After dedup, each expression belongs to exactly one TopN
+ // (the outermost one that can pull it up). The project needs to be
+ // simplified by removing all of them, exposing their base slots once.
+ List<NamedExpression> allPulledUpExprs = collectAllPulledUpExprs(context, rewritten);
+ if (allPulledUpExprs.isEmpty() && rewritten != project
+ && rewritten.getProjects().equals(project.getProjects())) {
+ allPulledUpExprs = collectAllPulledUpExprs(context, project);
+ }
+ return simplifyProject(rewritten, allPulledUpExprs, context);
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ LogicalTopN rewritten = (LogicalTopN) visit(topN, context);
+ PullUpInfo info = context.getPullUpInfo(rewritten);
+ if (info == null && rewritten != topN) {
+ info = context.getPullUpInfo(topN);
+ }
+ if (info == null || (info.allPulledUpExprs.isEmpty()
+ && info.passThroughExprByDeduplicatedExpr.isEmpty())) {
+ return rewritten;
+ }
+ return addUpperProject(rewritten, info, context);
+ }
+ }
+
+ /**
+ * Collect all pulled-up expressions across all PullUpInfos for a project.
+ * After dedup each expression belongs to exactly one TopN, but the project
+ * must be simplified by removing all of them at once.
+ */
+ private static List<NamedExpression> collectAllPulledUpExprs(
+ CollectorContext context, LogicalProject<?> project) {
+ List<NamedExpression> result = new ArrayList<>();
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> exprs = info.projectToPulledUpExprs.get(project);
+ if (exprs != null) {
+ result.addAll(exprs);
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Remove pulled-up expressions from this Project and add the input slots that still need to pass through TopN.
+ *
+ * <p>For example, after pulling up {@code x = a + 1}:
+ *
+ * <pre>
+ * TopN
+ * Project(id, x) -- forwards x from its child
+ * Project(id, a + 1 as x)
+ * Scan(id, a)
+ * </pre>
+ *
+ * <p>The lower Project should become {@code Project(id, a)}, because {@code x} is restored above TopN.
+ * The upper Project must also become {@code Project(id, a)} instead of keeping {@code Project(id, x)},
+ * since its child no longer outputs {@code x}.
+ */
+ private static LogicalProject<? extends Plan> simplifyProject(
+ LogicalProject<? extends Plan> project,
+ List<NamedExpression> pulledUpExprs,
+ CollectorContext context) {
+ Set<ExprId> childOutputExprIds = buildOutputExprIds((Plan) project.child(0));
+ List<Expression> passThroughExprs = collectUnavailablePullUpExprs(project, context, childOutputExprIds);
+ if (pulledUpExprs.isEmpty() && passThroughExprs.isEmpty()) {
+ return project;
+ }
+
+ Set<ExprId> pulledUpExprIds = new HashSet<>();
+ for (NamedExpression ne : pulledUpExprs) {
+ pulledUpExprIds.add(ne.getExprId());
+ }
+
+ List<NamedExpression> simplified = new ArrayList<>();
+ Set<ExprId> existingExprIds = new HashSet<>();
+ for (NamedExpression ne : project.getProjects()) {
+ if (!pulledUpExprIds.contains(ne.getExprId())
+ && !isUnavailablePullUpSlot(ne, context, childOutputExprIds)) {
+ NamedExpression resolved = resolveNamedExpression(ne, context, childOutputExprIds);
+ simplified.add(resolved);
+ existingExprIds.add(resolved.getExprId());
+ }
+ }
+
+ for (NamedExpression pulledUpExpr : pulledUpExprs) {
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ if (info.baseSlotsByExpr.get(pulledUpExpr.getExprId()) != null) {
+ for (Slot baseSlot : resolveInputSlots(pulledUpExpr, context, childOutputExprIds)) {
+ if (!existingExprIds.contains(baseSlot.getExprId())) {
+ simplified.add(baseSlot);
+ existingExprIds.add(baseSlot.getExprId());
+ }
+ }
+ break; // found, no need to check other PullUpInfos
+ }
+ }
+ }
+ for (Expression passThroughExpr : passThroughExprs) {
+ for (Slot baseSlot : resolveInputSlots(passThroughExpr, context, childOutputExprIds)) {
+ if (!existingExprIds.contains(baseSlot.getExprId())) {
+ simplified.add(baseSlot);
+ existingExprIds.add(baseSlot.getExprId());
+ }
+ }
+ }
+
+ if (simplified.equals(project.getProjects())) {
+ return project;
+ }
+ return (LogicalProject<? extends Plan>) project.withProjects(simplified);
+ }
+
+ private static List<Expression> collectUnavailablePullUpExprs(
+ LogicalProject<? extends Plan> project, CollectorContext context, Set<ExprId> childOutputExprIds) {
+ List<Expression> passThroughExprs = new ArrayList<>();
+ for (NamedExpression ne : project.getProjects()) {
+ if (isUnavailablePullUpSlot(ne, context, childOutputExprIds)) {
+ passThroughExprs.add(getPullUpReplaceExpression((Slot) ne, context));
+ }
+ }
+ return passThroughExprs;
+ }
+
+ private static boolean isUnavailablePullUpSlot(
+ NamedExpression ne, CollectorContext context, Set<ExprId> childOutputExprIds) {
+ return ne instanceof Slot
+ && !childOutputExprIds.contains(ne.getExprId())
+ && getPullUpReplaceExpression((Slot) ne, context) != null;
+ }
+
+ private static Expression getPullUpReplaceExpression(Slot slot, CollectorContext context) {
+ for (Map.Entry<Slot, Expression> entry : context.pullUpExprReplaceMap.entrySet()) {
+ if (entry.getKey().getExprId().equals(slot.getExprId())) {
+ return entry.getValue();
+ }
+ }
+ return null;
+ }
+
+ /** Create a new Project above the TopN that restores pulled-up expressions. */
+ private static LogicalProject<Plan> addUpperProject(LogicalTopN topN, PullUpInfo info,
+ CollectorContext context) {
+ Map<ExprId, NamedExpression> pulledUpBySlotExprId = new HashMap<>();
+ Set<ExprId> currentOutputExprIds = buildOutputExprIds(topN);
+ for (NamedExpression e : info.allPulledUpExprs) {
+ pulledUpBySlotExprId.put(e.toSlot().getExprId(), resolvePulledUpExpr(e, context, currentOutputExprIds));
+ }
+
+ // Use the current (possibly rewritten) TopN's output so that slots
+ // whose expressions were deduplicated to an outer TopN reference
+ // the correct post-simplification ExprIds instead of stale ones.
+ List<Slot> currentOutput = topN.getOutput();
+ Map<ExprId, Slot> currentOutputByExprId = new HashMap<>();
+ for (Slot slot : currentOutput) {
+ currentOutputByExprId.put(slot.getExprId(), slot);
+ }
+ List<NamedExpression> upperOutput = new ArrayList<>();
+ Set<ExprId> upperOutputExprIds = new HashSet<>();
+ Set<ExprId> passThroughOutputExprIds = new HashSet<>();
+ for (int i = 0; i < info.originalTopNOutput.size(); i++) {
+ Slot origSlot = info.originalTopNOutput.get(i);
+ NamedExpression pulledUpExpr = pulledUpBySlotExprId.get(origSlot.getExprId());
+ if (pulledUpExpr != null) {
+ upperOutput.add(pulledUpExpr);
+ upperOutputExprIds.add(pulledUpExpr.getExprId());
+ addPassThroughSlots(upperOutput, upperOutputExprIds, passThroughOutputExprIds,
Review Comment:
review11
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4624119770
<details>
<summary>TPC-H: <b>Total hot run time: 29114 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c005743605d8bb37f81400c7996fe80d09a8875, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17990 4011 3994 3994
q2 q3 10760 1439 792 792
q4 4689 474 343 343
q5 7659 863 581 581
q6 186 174 141 141
q7 764 850 635 635
q8 9408 1612 1620 1612
q9 5828 4532 4459 4459
q10 6760 1871 1529 1529
q11 427 273 247 247
q12 625 426 288 288
q13 18170 3444 2763 2763
q14 266 254 253 253
q15 q16 818 768 710 710
q17 895 921 966 921
q18 6859 5717 5582 5582
q19 1429 1326 1026 1026
q20 520 399 262 262
q21 6474 2857 2645 2645
q22 458 382 331 331
Total cold run time: 100985 ms
Total hot run time: 29114 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5132 4784 4718 4718
q2 q3 4976 5273 4808 4808
q4 2112 2169 1422 1422
q5 4801 4872 4702 4702
q6 228 176 131 131
q7 1860 1780 1523 1523
q8 2419 2097 2146 2097
q9 7896 7546 7375 7375
q10 4756 4681 4243 4243
q11 546 387 359 359
q12 729 743 525 525
q13 2997 3365 2839 2839
q14 292 280 254 254
q15 q16 670 704 619 619
q17 1269 1249 1249 1249
q18 7474 6808 6798 6798
q19 1121 1137 1090 1090
q20 2217 2204 1933 1933
q21 5269 4526 4413 4413
q22 496 457 422 422
Total cold run time: 57260 ms
Total hot run time: 51520 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4628443537
# FE Regression Coverage Report
Increment line coverage ` 78.99% (391/495)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_afba38054f35638e61af5da67ffb46085f0b55e7_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_afba38054f35638e61af5da67ffb46085f0b55e7_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4633724446
<details>
<summary>TPC-DS: <b>Total hot run time: 169359 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9019aef33ce3081fa110c40f4cc67cb0236c0556, data reload: false
query5 4317 625 470 470
query6 453 199 174 174
query7 4847 570 305 305
query8 367 211 208 208
query9 8768 4073 4055 4055
query10 449 343 260 260
query11 5940 2330 2138 2138
query12 158 103 97 97
query13 1251 624 441 441
query14 6400 5430 5055 5055
query14_1 4393 4395 4405 4395
query15 208 204 184 184
query16 1000 472 439 439
query17 1133 737 597 597
query18 2479 502 354 354
query19 212 188 149 149
query20 118 111 109 109
query21 221 146 122 122
query22 13605 13556 13503 13503
query23 17395 16552 16122 16122
query23_1 16254 16361 16304 16304
query24 7404 1770 1319 1319
query24_1 1304 1349 1307 1307
query25 584 478 419 419
query26 1307 337 172 172
query27 2647 557 356 356
query28 4470 1996 2046 1996
query29 1086 640 531 531
query30 333 247 202 202
query31 1121 1086 966 966
query32 107 64 61 61
query33 555 335 276 276
query34 1176 1148 659 659
query35 765 816 677 677
query36 1389 1422 1223 1223
query37 158 107 85 85
query38 3220 3188 3093 3093
query39 940 922 888 888
query39_1 872 876 869 869
query40 219 122 100 100
query41 65 65 60 60
query42 93 93 96 93
query43 318 324 277 277
query44
query45 200 188 180 180
query46 1106 1184 775 775
query47 2354 2371 2197 2197
query48 406 384 298 298
query49 622 481 347 347
query50 995 350 247 247
query51 4465 4334 4280 4280
query52 89 90 77 77
query53 243 272 191 191
query54 261 231 194 194
query55 78 77 75 75
query56 224 230 237 230
query57 1433 1406 1328 1328
query58 242 222 211 211
query59 1541 1683 1420 1420
query60 295 254 231 231
query61 160 158 163 158
query62 690 653 588 588
query63 218 195 190 190
query64 2560 785 671 671
query65
query66 1811 465 341 341
query67 29658 29643 29472 29472
query68
query69 418 299 262 262
query70 998 966 893 893
query71 297 225 241 225
query72 3079 2737 2429 2429
query73 810 771 436 436
query74 5145 4940 4744 4744
query75 2673 2565 2270 2270
query76 2308 1160 786 786
query77 352 384 286 286
query78 12432 12714 11904 11904
query79 1455 1068 789 789
query80 595 490 402 402
query81 461 289 242 242
query82 570 156 121 121
query83 364 276 255 255
query84
query85 873 546 461 461
query86 371 304 270 270
query87 3372 3383 3167 3167
query88 3627 2735 2733 2733
query89 425 385 324 324
query90 1974 182 168 168
query91 180 163 144 144
query92 64 61 57 57
query93 1489 1521 881 881
query94 549 357 312 312
query95 709 457 343 343
query96 1115 813 343 343
query97 2697 2697 2605 2605
query98 211 209 205 205
query99 1174 1157 1030 1030
Total cold run time: 250846 ms
Total hot run time: 169359 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4598072313
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4625295383
# FE Regression Coverage Report
Increment line coverage ` 78.99% (391/495)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_9c005743605d8bb37f81400c7996fe80d09a8875_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_9c005743605d8bb37f81400c7996fe80d09a8875_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4621050885
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3355611667
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,710 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+ final Map<ExprId, NamedExpression> passThroughExprByDeduplicatedExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+
+ void addPassThroughExprForDeduplicatedExpr(NamedExpression expr) {
+ passThroughExprByDeduplicatedExpr.put(expr.getExprId(), expr);
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ /**
+ * Use IdentityHashMap so that two different TopN nodes with the same
+ * content (orderKeys, limit, offset) are treated as distinct keys.
+ * LogicalTopN.equals() is content-based, which would cause unrelated
+ * TopN nodes to collide in a regular HashMap/LinkedHashMap.
+ */
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new IdentityHashMap<>();
+ /**
+ * Maintain insertion order for deterministic outer-to-inner iteration
+ * in dedup and other passes. The Collector visits the plan top-down,
+ * so the order is naturally outer-before-inner.
+ */
+ final List<LogicalTopN> topNOrder = new ArrayList<>();
+ final Map<Slot, Expression> pullUpExprReplaceMap = new LinkedHashMap<>();
+ /**
+ * When collectFromNode encounters a nested TopN, it saves the current
+ * blockedExprIds (accumulated from outer nodes) so that visitLogicalTopN
+ * for the inner TopN can merge them into its fresh blocked set.
+ */
+ final Map<LogicalTopN, Set<ExprId>> outerBlockedByTopN = new IdentityHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ void addPullUpInfo(LogicalTopN topN, PullUpInfo info) {
+ topNToPullUpInfo.put(topN, info);
+ topNOrder.add(topN);
+ }
+
+ void addPullUpExprReplace(NamedExpression expr) {
+ if (expr instanceof Alias) {
+ pullUpExprReplaceMap.putIfAbsent(expr.toSlot(), expr.child(0));
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ // If this is a nested TopN, merge in the outer blocked set that was
+ // saved by collectFromNode when it encountered this TopN. This
+ // ensures that slots consumed by outer operators (e.g. join
+ // conditions above this TopN) also block pull-up from projects
+ // under this TopN.
+ Set<ExprId> outerBlocked = context.outerBlockedByTopN.remove(topN);
+ if (outerBlocked != null) {
+ blockedExprIds.addAll(outerBlocked);
+ }
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds, context);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ for (NamedExpression expr : info.allPulledUpExprs) {
+ context.addPullUpExprReplace(expr);
+ }
+ context.addPullUpInfo(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds,
+ CollectorContext context) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds, context);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // Save the current blockedExprIds (accumulated from outer nodes
+ // such as outer TopN + intermediate Joins) so that the inner
+ // TopN's own visitLogicalTopN can merge them into its fresh
+ // blocked set. Without this, outer join condition slots would
+ // not block pull-up from projects under the inner TopN.
+ context.outerBlockedByTopN.put(inner, new HashSet<>(blockedExprIds));
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked, context);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For null-generating outer joins, block all output slots from the
+ // nullable side(s). Expressions inside a nullable side are protected
+ // by join null-extension: when there is no match, the entire nullable
+ // tuple is set to NULL. Pulling such an expression above the join
+ // would break this, e.g. ifnull(r.b, 0) inside the right side of a
+ // LEFT JOIN would see individual column NULLs and convert them to 0,
+ // changing the NULL that null-extension produced.
+ // Example: SELECT l.id, sub.x FROM l LEFT JOIN (
+ // SELECT id, ifnull(b, 0) AS x FROM r) sub ON l.id = sub.id
+ // ORDER BY l.id LIMIT 3;
+ // Here x=ifnull(b,0) is in a Project on the nullable (right) side.
+ // Pulling it above the join turns unmatched-row x from NULL to 0.
+ if (node instanceof LogicalJoin) {
+ LogicalJoin<?, ?> join = (LogicalJoin<?, ?>) node;
+ JoinType joinType = join.getJoinType();
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ // add join expression slots (same as default branch)
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+ // block all output slots from the nullable side(s)
+ if (joinType.isLeftOuterJoin() || joinType.isFullOuterJoin()) {
Review Comment:
The nullable-side guard still misses ASOF outer joins. `ASOF_LEFT_OUTER_JOIN` also null-extends the right side, and `ASOF_RIGHT_OUTER_JOIN` null-extends the left side, but these checks only cover ordinary `LEFT_OUTER_JOIN`/`RIGHT_OUTER_JOIN`/`FULL_OUTER_JOIN`. A plan such as `TopN -> Project(l.ts, x) -> ASOF LEFT OUTER JOIN(left=l, right=Project(ifnull(r.v, 0) AS x, r.ts))` can still collect `x` from the right child; after simplification/restoration, unmatched rows evaluate `ifnull(NULL, 0)` above the join instead of preserving the null-extended `x = NULL`. This is distinct from the existing ordinary outer-join thread because the current fix does not include the ASOF outer join types. Please include `joinType.isAsofLeftOuterJoin()` in the right-side blocking condition and `joinType.isAsofRightOuterJoin()` in the left-side blocking condition, and add an ASOF outer join negative test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4613107433
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4629410010
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4630765726
# FE Regression Coverage Report
Increment line coverage ` 70.76% (392/554)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_90b0103ac670a6a7745e940394f4a283481be422_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_90b0103ac670a6a7745e940394f4a283481be422_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4633455451
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4565804539
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3310758931
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/materialize/LazyMaterializeTopN.java:
##########
@@ -92,29 +87,206 @@ private Plan computeTopN(PhysicalTopN topN, CascadesContext ctx) {
if (SessionVariable.getTopNLazyMaterializationThreshold() < topN.getLimit()) {
return topN;
}
- /*
- topn(output=[x] orderkey=[b])
- ->project(a as x)
- ->T(a, b)
- 'x' can be lazy materialized.
- materializeMap: x->(T, a)
- */
+ try {
+ List<Slot> userVisibleOutput = ImmutableList.copyOf(topN.getOutput());
+
+ // Find projects below TopN with PreferPushDownProject expressions.
+ // Simplify them and expose hidden columns as lazy candidates.
+ List<NamedExpression> pulledUpExprs = new ArrayList<>();
+ List<Slot> lazyCandidates = new ArrayList<>();
+ PhysicalProject<? extends Plan> leafProject = findLeafProject(topN);
+ boolean restructured = false;
+
+ if (leafProject != null && hasPreferPushDownProjectExprs(leafProject)) {
+ // Split: remove PPD/subPath expressions from lower project
+ List<NamedExpression> simplified = new ArrayList<>();
+ for (NamedExpression ne : leafProject.getProjects()) {
+ if (isNestedLazyExpression(ne)) {
+ pulledUpExprs.add(ne);
+ // For PPD expressions, input slots ARE the lazy candidates
+ // (e.g., struct_col for struct_element, payload['name'] for variant)
+ lazyCandidates.addAll(ne.getInputSlots());
+ } else {
+ simplified.add(ne);
+ }
+ }
+ // Add lazy candidate slots to lower project so LazySlotPruning can remove them
+ for (Slot slot : lazyCandidates) {
+ if (!containsSlotInExprs(simplified, slot)) {
+ simplified.add(slot);
+ }
+ }
+ topN = replaceLeafProject(topN, leafProject, simplified);
+ restructured = true;
+ }
+
+ // Build effective probe list: TopN output (excl. pulled-up slots) + lazy candidates
+ Set<ExprId> pulledUpExprIds = new HashSet<>();
+ for (NamedExpression ne : pulledUpExprs) {
+ pulledUpExprIds.add(ne.getExprId());
+ }
+ List<Slot> effectiveOutput = new ArrayList<>();
+ for (Slot slot : topN.getOutput()) {
+ if (!pulledUpExprIds.contains(slot.getExprId())) {
+ effectiveOutput.add(slot);
+ }
+ }
+ if (restructured) {
+ for (Slot candidate : lazyCandidates) {
+ if (!containsSlot(effectiveOutput, candidate)) {
+ effectiveOutput.add(candidate);
+ }
+ }
+ }
+
+ Plan result = doComputeTopN(topN, ctx, ImmutableList.copyOf(effectiveOutput),
+ restructured ? leafProject : null, lazyCandidates);
+
+ if (pulledUpExprs.isEmpty()) {
+ result = new PhysicalProject(ImmutableList.copyOf(userVisibleOutput), null, result);
+ } else {
+ List<NamedExpression> outputExprs = new ArrayList<>();
+ for (Slot slot : userVisibleOutput) {
+ if (!pulledUpExprIds.contains(slot.getExprId())) {
+ outputExprs.add(slot);
+ }
+ }
+ outputExprs.addAll(pulledUpExprs);
+ result = new PhysicalProject(ImmutableList.copyOf(outputExprs), null, result);
Review Comment:
This rebuilds the final projection as all non-pulled slots followed by all pulled-up expressions, which changes the user-visible column order whenever a pulled-up nested expression is not already the last selected column. For example `select substring(struct_element(struct_col, 'city'), 1) as city, id ... order by id limit 3` has `userVisibleOutput = [city, id]`, but this code produces `[id, city]`, so the result schema and row values are swapped relative to the SQL. Please preserve the original `userVisibleOutput` order by replacing each pulled slot in-place with its corresponding pulled-up expression, and add a regression case with the nested expression before another selected column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3337791676
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,452 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked);
+ }
+ }
+
+ // =========================================================================
+ // Pull-up eligibility
+ // =========================================================================
+
+ /**
+ * Check if a named expression can be pulled up above TopN.
+ * Eligible: Alias with non-trivial child, not blocked, no NoneMovableFunction.
+ */
+ static boolean canPullUp(NamedExpression ne) {
+ if (!(ne instanceof Alias)) {
+ return false;
+ }
+ Expression child = ((Alias) ne).child();
+ if (child instanceof Slot || child instanceof Literal) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof NoneMovableFunction)) {
+ return false;
+ }
+ return true;
+ }
+
+ private static boolean isBlockingNode(Plan node) {
+ return node instanceof LogicalAggregate
+ || node instanceof LogicalWindow
+ || node instanceof LogicalRepeat;
+ }
+
+ private static Set<ExprId> buildOrderKeyExprIds(LogicalTopN<?> topN) {
+ Set<ExprId> orderKeyExprIds = new HashSet<>();
+ for (OrderKey orderKey : topN.getOrderKeys()) {
+ Expression keyExpr = orderKey.getExpr();
+ orderKeyExprIds.addAll(keyExpr.getInputSlotExprIds());
+ if (keyExpr instanceof NamedExpression) {
+ orderKeyExprIds.add(((NamedExpression) keyExpr).getExprId());
+ }
+ }
+ return orderKeyExprIds;
+ }
+
+ /**
+ * Deduplicate pull-up expressions so that each expression in a Project is only
+ * pulled up to the outermost TopN that collects it.
+ *
+ * <p>Since {@link CollectorContext#topNToPullUpInfo} is a {@link LinkedHashMap}
+ * and the Collector visits the plan top-down, iteration order is outer-to-inner.
+ * We keep the first occurrence of each (project-reference, exprId) pair and
+ * remove duplicates from inner TopNs.
+ */
+ private static void deduplicatePullUps(CollectorContext context) {
+ // Use IdentityHashMap because we need to distinguish Project nodes by object
+ // reference, not by content equality.
+ Map<LogicalProject<? extends Plan>, Set<ExprId>> handled = new IdentityHashMap<>();
+
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> toRemove = new ArrayList<>();
+ for (Map.Entry<LogicalProject<? extends Plan>, List<NamedExpression>> entry
+ : info.projectToPulledUpExprs.entrySet()) {
+ LogicalProject<? extends Plan> project = entry.getKey();
+ Set<ExprId> projectHandled = handled.computeIfAbsent(project, k -> new HashSet<>());
+ for (NamedExpression expr : entry.getValue()) {
+ if (projectHandled.contains(expr.getExprId())) {
+ toRemove.add(expr);
+ } else {
+ projectHandled.add(expr.getExprId());
+ }
+ }
+ }
+ for (NamedExpression expr : toRemove) {
+ info.allPulledUpExprs.remove(expr);
+ for (List<NamedExpression> list : info.projectToPulledUpExprs.values()) {
+ list.removeIf(e -> e == expr);
+ }
+ info.projectToPulledUpExprs.entrySet().removeIf(e -> e.getValue().isEmpty());
+ info.baseSlotsByExpr.remove(expr.getExprId());
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 2: Replacer (bottom-up)
+ // =========================================================================
+
+ static class Replacer extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, CollectorContext context) {
+ LogicalProject<? extends Plan> rewritten = (LogicalProject<? extends Plan>) visit(project, context);
+
+ // Collect ALL pulled-up expressions across ALL PullUpInfos for this
+ // project. After dedup, each expression belongs to exactly one TopN
+ // (the outermost one that can pull it up). The project needs to be
+ // simplified by removing all of them, exposing their base slots once.
+ List<NamedExpression> allPulledUpExprs = collectAllPulledUpExprs(context, rewritten);
+ if (allPulledUpExprs.isEmpty() && rewritten != project
+ && rewritten.getProjects().equals(project.getProjects())) {
+ allPulledUpExprs = collectAllPulledUpExprs(context, project);
+ }
+ if (allPulledUpExprs.isEmpty()) {
+ return rewritten;
+ }
+ return simplifyProject(rewritten, allPulledUpExprs, context);
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ LogicalTopN rewritten = (LogicalTopN) visit(topN, context);
+ PullUpInfo info = context.getPullUpInfo(rewritten);
+ if (info == null && rewritten != topN) {
+ info = context.getPullUpInfo(topN);
+ }
+ if (info == null || info.allPulledUpExprs.isEmpty()) {
+ return rewritten;
+ }
+ return addUpperProject(rewritten, info);
+ }
+ }
+
+ /**
+ * Collect all pulled-up expressions across all PullUpInfos for a project.
+ * After dedup each expression belongs to exactly one TopN, but the project
+ * must be simplified by removing all of them at once.
+ */
+ private static List<NamedExpression> collectAllPulledUpExprs(
+ CollectorContext context, LogicalProject<?> project) {
+ List<NamedExpression> result = new ArrayList<>();
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> exprs = info.projectToPulledUpExprs.get(project);
+ if (exprs != null) {
+ result.addAll(exprs);
+ }
+ }
+ return result;
+ }
+
+ /** Remove pulled-up expressions from project and add their base input slots. */
+ private static LogicalProject<? extends Plan> simplifyProject(
+ LogicalProject<? extends Plan> project,
+ List<NamedExpression> pulledUpExprs,
+ CollectorContext context) {
+ if (pulledUpExprs.isEmpty()) {
+ return project;
+ }
+
+ Set<ExprId> pulledUpExprIds = new HashSet<>();
+ for (NamedExpression ne : pulledUpExprs) {
+ pulledUpExprIds.add(ne.getExprId());
+ }
+
+ List<NamedExpression> simplified = new ArrayList<>();
+ Set<ExprId> existingExprIds = new HashSet<>();
+ for (NamedExpression ne : project.getProjects()) {
+ if (!pulledUpExprIds.contains(ne.getExprId())) {
+ simplified.add(ne);
+ existingExprIds.add(ne.getExprId());
+ }
+ }
+
+ for (NamedExpression pulledUpExpr : pulledUpExprs) {
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<Slot> baseSlots = info.baseSlotsByExpr.get(pulledUpExpr.getExprId());
+ if (baseSlots != null) {
+ for (Slot baseSlot : baseSlots) {
+ if (!existingExprIds.contains(baseSlot.getExprId())) {
+ simplified.add(baseSlot);
+ existingExprIds.add(baseSlot.getExprId());
+ }
+ }
+ break; // found, no need to check other PullUpInfos
+ }
+ }
+ }
+
+ if (simplified.equals(project.getProjects())) {
+ return project;
+ }
+ return (LogicalProject<? extends Plan>) project.withProjects(simplified);
+ }
+
+ /** Create a new Project above the TopN that restores pulled-up expressions. */
+ private static LogicalProject<Plan> addUpperProject(LogicalTopN topN, PullUpInfo info) {
+ Map<ExprId, NamedExpression> pulledUpBySlotExprId = new HashMap<>();
+ for (NamedExpression e : info.allPulledUpExprs) {
+ pulledUpBySlotExprId.put(e.toSlot().getExprId(), e);
+ }
+
+ // Use the current (possibly rewritten) TopN's output so that slots
+ // whose expressions were deduplicated to an outer TopN reference
+ // the correct post-simplification ExprIds instead of stale ones.
+ List<Slot> currentOutput = topN.getOutput();
+ List<NamedExpression> upperOutput = new ArrayList<>();
+ for (int i = 0; i < info.originalTopNOutput.size(); i++) {
+ Slot origSlot = info.originalTopNOutput.get(i);
+ NamedExpression pulledUpExpr = pulledUpBySlotExprId.get(origSlot.getExprId());
+ if (pulledUpExpr != null) {
+ upperOutput.add(pulledUpExpr);
+ } else if (i < currentOutput.size()) {
+ // Slot was not pulled up: use the current slot at the same
+ // position so the ExprId matches the rewritten child subtree.
+ upperOutput.add(currentOutput.get(i));
Review Comment:
review6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4590930286
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4590930945
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4600121004
<details>
<summary>TPC-DS: <b>Total hot run time: 168992 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0862cc18bf424da86e16acff60787158a0febcd, data reload: false
query5 4347 634 507 507
query6 455 198 186 186
query7 4824 584 320 320
query8 372 220 209 209
query9 8755 3991 4025 3991
query10 448 315 261 261
query11 5994 2434 2205 2205
query12 165 105 104 104
query13 1245 626 443 443
query14 6261 5451 5078 5078
query14_1 4398 4417 4400 4400
query15 207 195 180 180
query16 1023 466 429 429
query17 1132 722 606 606
query18 2722 485 359 359
query19 215 188 151 151
query20 117 110 105 105
query21 226 139 121 121
query22 13703 13547 13345 13345
query23 17384 16427 16145 16145
query23_1 16305 16205 16326 16205
query24 7633 1785 1293 1293
query24_1 1314 1330 1306 1306
query25 588 466 402 402
query26 1311 316 178 178
query27 2654 580 333 333
query28 4448 2062 2036 2036
query29 1093 636 505 505
query30 313 237 200 200
query31 1129 1080 958 958
query32 114 63 61 61
query33 542 335 265 265
query34 1185 1154 658 658
query35 785 787 717 717
query36 1364 1384 1210 1210
query37 155 104 90 90
query38 3264 3123 3105 3105
query39 925 924 914 914
query39_1 878 867 889 867
query40 216 125 99 99
query41 64 63 66 63
query42 96 95 94 94
query43 319 318 277 277
query44
query45 198 182 176 176
query46 1118 1282 742 742
query47 2344 2357 2254 2254
query48 406 404 304 304
query49 623 468 347 347
query50 1014 346 253 253
query51 4335 4262 4205 4205
query52 88 90 77 77
query53 241 280 197 197
query54 266 227 198 198
query55 79 81 71 71
query56 249 232 231 231
query57 1429 1394 1283 1283
query58 254 215 206 206
query59 1558 1679 1388 1388
query60 298 242 232 232
query61 159 158 158 158
query62 691 642 588 588
query63 228 192 187 187
query64 2530 781 645 645
query65
query66 1766 459 338 338
query67 29159 29696 29558 29558
query68
query69 436 305 277 277
query70 921 941 979 941
query71 330 220 220 220
query72 2997 2682 2380 2380
query73 851 746 434 434
query74 5149 4972 4772 4772
query75 2679 2578 2262 2262
query76 2346 1156 762 762
query77 361 382 296 296
query78 12420 12436 11892 11892
query79 1464 1119 784 784
query80 1322 468 409 409
query81 525 278 244 244
query82 637 161 118 118
query83 343 279 251 251
query84
query85 927 560 452 452
query86 420 291 286 286
query87 3378 3415 3170 3170
query88 3742 2804 2740 2740
query89 425 386 340 340
query90 1903 189 185 185
query91 180 164 140 140
query92 64 70 56 56
query93 1630 1486 895 895
query94 708 360 303 303
query95 706 472 364 364
query96 1049 812 347 347
query97 2703 2708 2544 2544
query98 215 207 203 203
query99 1137 1186 1034 1034
Total cold run time: 252103 ms
Total hot run time: 168992 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4622268448
<details>
<summary>TPC-DS: <b>Total hot run time: 169432 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa859dd138717a39731d64ec330321c62f5f4cc4, data reload: false
query5 4338 628 482 482
query6 440 202 177 177
query7 4845 554 316 316
query8 380 236 214 214
query9 8762 4126 4084 4084
query10 458 317 270 270
query11 5975 2336 2129 2129
query12 163 107 101 101
query13 1307 626 444 444
query14 6464 5408 5087 5087
query14_1 4416 4423 4392 4392
query15 210 207 177 177
query16 985 464 419 419
query17 927 693 584 584
query18 2447 458 342 342
query19 199 184 141 141
query20 112 108 103 103
query21 214 141 115 115
query22 13663 13516 13515 13515
query23 17119 16483 16122 16122
query23_1 16258 16351 16321 16321
query24 7454 1779 1319 1319
query24_1 1345 1332 1346 1332
query25 590 478 424 424
query26 1299 316 174 174
query27 2667 579 345 345
query28 4470 2060 2073 2060
query29 1130 691 515 515
query30 304 243 202 202
query31 1152 1072 958 958
query32 117 64 61 61
query33 545 325 274 274
query34 1196 1107 653 653
query35 755 804 694 694
query36 1383 1381 1241 1241
query37 156 110 96 96
query38 3227 3145 3073 3073
query39 929 911 921 911
query39_1 890 890 904 890
query40 227 129 107 107
query41 72 73 83 73
query42 103 98 98 98
query43 321 334 287 287
query44
query45 201 193 187 187
query46 1090 1183 735 735
query47 2367 2373 2215 2215
query48 403 426 287 287
query49 641 484 376 376
query50 1005 361 269 269
query51 4377 4304 4330 4304
query52 90 93 80 80
query53 254 273 198 198
query54 287 244 216 216
query55 82 78 73 73
query56 248 237 240 237
query57 1454 1412 1346 1346
query58 253 232 227 227
query59 1588 1676 1461 1461
query60 302 268 246 246
query61 210 159 155 155
query62 715 656 587 587
query63 230 192 194 192
query64 2584 775 633 633
query65
query66 1791 462 369 369
query67 29727 29153 29554 29153
query68
query69 418 303 257 257
query70 1004 948 960 948
query71 302 218 218 218
query72 2957 2714 2357 2357
query73 877 768 440 440
query74 5132 4941 4779 4779
query75 2678 2574 2259 2259
query76 2324 1177 815 815
query77 364 398 290 290
query78 12357 12390 11924 11924
query79 1443 1058 755 755
query80 592 478 407 407
query81 455 275 255 255
query82 572 159 119 119
query83 385 279 248 248
query84
query85 886 534 471 471
query86 364 298 293 293
query87 3417 3414 3235 3235
query88 3710 2787 2763 2763
query89 434 384 334 334
query90 1985 187 184 184
query91 180 160 135 135
query92 62 64 56 56
query93 1523 1487 850 850
query94 546 347 330 330
query95 690 479 358 358
query96 1078 847 336 336
query97 2688 2732 2554 2554
query98 212 208 222 208
query99 1150 1198 1026 1026
Total cold run time: 251129 ms
Total hot run time: 169432 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4634672439
# FE Regression Coverage Report
Increment line coverage ` 78.64% (394/501)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_9019aef33ce3081fa110c40f4cc67cb0236c0556_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_9019aef33ce3081fa110c40f4cc67cb0236c0556_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4604390215
<details>
<summary>TPC-DS: <b>Total hot run time: 169019 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 14da24479173944ea5601dd5c550fa43e7079e0d, data reload: false
query5 4321 630 486 486
query6 457 201 181 181
query7 4827 595 300 300
query8 374 219 203 203
query9 8745 3969 3986 3969
query10 453 310 261 261
query11 5867 2350 2147 2147
query12 161 104 108 104
query13 1264 601 416 416
query14 6436 5429 5094 5094
query14_1 4399 4372 4395 4372
query15 211 198 176 176
query16 1006 471 449 449
query17 946 710 589 589
query18 2469 481 352 352
query19 206 187 146 146
query20 116 108 110 108
query21 221 142 117 117
query22 13586 13643 13421 13421
query23 17218 16419 16183 16183
query23_1 16364 16248 16232 16232
query24 7735 1732 1319 1319
query24_1 1329 1296 1306 1296
query25 570 446 378 378
query26 1316 326 161 161
query27 2672 574 356 356
query28 4447 2026 1974 1974
query29 1083 592 478 478
query30 310 240 205 205
query31 1117 1082 938 938
query32 110 61 62 61
query33 517 315 259 259
query34 1162 1143 649 649
query35 750 789 685 685
query36 1403 1432 1245 1245
query37 154 105 89 89
query38 3212 3151 3017 3017
query39 945 933 915 915
query39_1 892 868 869 868
query40 234 131 102 102
query41 67 62 64 62
query42 96 95 92 92
query43 315 319 280 280
query44
query45 196 185 178 178
query46 1074 1193 712 712
query47 2379 2375 2249 2249
query48 396 431 310 310
query49 630 467 362 362
query50 957 334 252 252
query51 4337 4265 4219 4219
query52 88 87 76 76
query53 238 261 191 191
query54 264 235 202 202
query55 78 75 69 69
query56 262 218 212 212
query57 1420 1414 1330 1330
query58 251 217 210 210
query59 1520 1619 1426 1426
query60 290 252 233 233
query61 174 164 168 164
query62 711 648 605 605
query63 239 188 191 188
query64 2593 787 619 619
query65
query66 1844 478 348 348
query67 29714 29731 29511 29511
query68
query69 426 300 268 268
query70 972 983 912 912
query71 298 229 208 208
query72 3090 2648 2379 2379
query73 889 747 436 436
query74 5119 4951 4742 4742
query75 2662 2595 2277 2277
query76 2328 1186 799 799
query77 362 372 296 296
query78 12380 12368 11880 11880
query79 1301 1027 733 733
query80 617 486 394 394
query81 455 278 239 239
query82 589 155 119 119
query83 363 277 247 247
query84
query85 881 533 439 439
query86 373 294 283 283
query87 3387 3371 3215 3215
query88 3639 2752 2695 2695
query89 416 381 329 329
query90 1931 181 173 173
query91 185 163 139 139
query92 62 58 58 58
query93 1490 1412 866 866
query94 536 353 318 318
query95 671 483 345 345
query96 1043 822 330 330
query97 2745 2723 2559 2559
query98 220 205 204 204
query99 1162 1178 1038 1038
Total cold run time: 250614 ms
Total hot run time: 169019 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4554393696
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4566031479
<details>
<summary>TPC-H: <b>Total hot run time: 31458 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 835b604c4ad63b88f63c6caeb098d12c98a4be08, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17711 4086 4003 4003
q2 q3 10756 1372 803 803
q4 4690 478 347 347
q5 7667 2294 2086 2086
q6 241 179 138 138
q7 942 797 643 643
q8 9370 1681 1654 1654
q9 5198 4916 4913 4913
q10 6383 2206 1879 1879
q11 441 269 242 242
q12 636 428 298 298
q13 18111 3373 2774 2774
q14 261 257 238 238
q15 q16 816 767 710 710
q17 907 957 970 957
q18 6898 5728 5506 5506
q19 1320 1248 970 970
q20 521 511 345 345
q21 6190 2745 2626 2626
q22 457 377 326 326
Total cold run time: 99516 ms
Total hot run time: 31458 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5011 4698 4737 4698
q2 q3 4778 5307 4657 4657
q4 2147 2189 1394 1394
q5 4850 4915 4683 4683
q6 227 182 127 127
q7 1791 1744 1639 1639
q8 2427 2103 2111 2103
q9 7968 7987 7350 7350
q10 4745 4703 4198 4198
q11 526 383 349 349
q12 738 733 524 524
q13 3036 3439 2784 2784
q14 273 283 265 265
q15 q16 699 693 610 610
q17 1273 1249 1250 1249
q18 7201 6753 6771 6753
q19 1148 1106 1067 1067
q20 2205 2226 1948 1948
q21 5250 4561 4404 4404
q22 508 459 413 413
Total cold run time: 56801 ms
Total hot run time: 51215 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4577355577
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4613261725
# FE UT Coverage Report
Increment line coverage ` 72.65% (340/468)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_9f5ae1723eb5656a68b6b59e0c4848ed261c931c/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_9f5ae1723eb5656a68b6b59e0c4848ed261c931c/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4627697673
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4627954132
# FE UT Coverage Report
Increment line coverage ` 67.88% (336/495)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_afba38054f35638e61af5da67ffb46085f0b55e7/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_afba38054f35638e61af5da67ffb46085f0b55e7/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4613011731
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4555496478
<details>
<summary>TPC-DS: <b>Total hot run time: 172873 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4897f0cfc299fd90b64a17d6658e5891a6bbc55b, data reload: false
query5 4319 661 547 547
query6 338 233 199 199
query7 4305 539 317 317
query8 334 269 221 221
query9 8791 4119 4096 4096
query10 474 349 299 299
query11 5826 2537 2304 2304
query12 191 131 124 124
query13 1273 597 462 462
query14 6141 5511 5259 5259
query14_1 4571 4539 4581 4539
query15 219 215 186 186
query16 1034 469 442 442
query17 1155 760 637 637
query18 2701 500 377 377
query19 227 213 175 175
query20 145 137 135 135
query21 219 143 124 124
query22 13670 13655 13325 13325
query23 17531 16437 16206 16206
query23_1 16308 16462 16496 16462
query24 7426 1771 1371 1371
query24_1 1320 1350 1341 1341
query25 573 506 454 454
query26 1310 331 178 178
query27 2663 585 345 345
query28 4436 2059 2051 2051
query29 1013 662 515 515
query30 307 247 213 213
query31 1134 1079 955 955
query32 96 79 76 76
query33 557 357 336 336
query34 1241 1146 651 651
query35 785 783 715 715
query36 1396 1375 1224 1224
query37 154 108 90 90
query38 3217 3169 3096 3096
query39 934 923 902 902
query39_1 878 886 876 876
query40 228 146 125 125
query41 65 64 62 62
query42 114 116 108 108
query43 342 334 296 296
query44
query45 209 209 199 199
query46 1071 1210 742 742
query47 2367 2333 2213 2213
query48 383 425 295 295
query49 638 497 412 412
query50 980 361 257 257
query51 4372 4388 4253 4253
query52 107 104 98 98
query53 258 284 209 209
query54 325 267 258 258
query55 95 91 90 90
query56 311 328 334 328
query57 1441 1421 1298 1298
query58 298 270 280 270
query59 1603 1663 1462 1462
query60 334 319 312 312
query61 155 151 155 151
query62 706 645 592 592
query63 252 205 211 205
query64 2397 842 622 622
query65
query66 1678 492 360 360
query67 29246 29739 29545 29545
query68
query69 473 344 310 310
query70 1000 1059 1069 1059
query71 306 284 275 275
query72 3048 2768 2420 2420
query73 890 788 435 435
query74 5111 4993 4809 4809
query75 2707 2615 2264 2264
query76 2274 1169 764 764
query77 414 420 343 343
query78 12467 12492 11823 11823
query79 1446 1004 761 761
query80 687 539 454 454
query81 467 282 240 240
query82 1374 157 124 124
query83 328 271 252 252
query84 263 142 115 115
query85 903 538 452 452
query86 392 341 336 336
query87 3454 3392 3256 3256
query88 3629 2761 2749 2749
query89 455 393 345 345
query90 1933 186 191 186
query91 186 170 141 141
query92 80 74 79 74
query93 1562 1422 830 830
query94 555 364 300 300
query95 670 383 437 383
query96 1117 794 375 375
query97 2744 2746 2606 2606
query98 235 229 226 226
query99 1181 1154 1041 1041
Total cold run time: 254570 ms
Total hot run time: 172873 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4577356389
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4598071778
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4604243239
<details>
<summary>TPC-H: <b>Total hot run time: 29408 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17045ff0b7bfb226e39b0e057cf8c19fdd0947e4, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17594 4208 4266 4208
q2 q3 10775 1397 808 808
q4 4689 481 340 340
q5 7633 875 587 587
q6 176 171 138 138
q7 774 850 628 628
q8 9351 1536 1598 1536
q9 5789 4493 4534 4493
q10 6821 1787 1550 1550
q11 434 277 250 250
q12 625 437 289 289
q13 18146 3394 2733 2733
q14 270 255 247 247
q15 q16 803 787 704 704
q17 944 965 1000 965
q18 6811 5727 5560 5560
q19 1318 1289 1076 1076
q20 532 404 256 256
q21 6217 2836 2727 2727
q22 458 380 313 313
Total cold run time: 100160 ms
Total hot run time: 29408 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5072 4791 4823 4791
q2 q3 4903 5307 4708 4708
q4 2103 2200 1383 1383
q5 4800 4946 4739 4739
q6 239 178 129 129
q7 1871 1758 1607 1607
q8 2435 2085 2073 2073
q9 7973 7776 7447 7447
q10 4751 4655 4253 4253
q11 532 383 345 345
q12 728 733 527 527
q13 2983 3393 2841 2841
q14 268 276 252 252
q15 q16 675 697 600 600
q17 1264 1239 1235 1235
q18 7312 7061 6862 6862
q19 1119 1086 1105 1086
q20 2211 2216 1939 1939
q21 5258 4549 4420 4420
q22 514 488 399 399
Total cold run time: 57011 ms
Total hot run time: 51636 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4602926048
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4604332192
<details>
<summary>TPC-DS: <b>Total hot run time: 169813 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17045ff0b7bfb226e39b0e057cf8c19fdd0947e4, data reload: false
query5 4325 670 478 478
query6 444 198 177 177
query7 4922 576 304 304
query8 379 225 193 193
query9 8766 3966 3987 3966
query10 467 303 265 265
query11 5972 2365 2188 2188
query12 157 103 97 97
query13 1275 553 431 431
query14 6407 5364 5075 5075
query14_1 4373 4386 4368 4368
query15 210 194 174 174
query16 992 452 412 412
query17 1056 689 563 563
query18 2460 473 336 336
query19 208 182 137 137
query20 113 107 105 105
query21 223 138 117 117
query22 13790 13619 13389 13389
query23 17499 16574 16146 16146
query23_1 16369 16345 16306 16306
query24 7560 1762 1298 1298
query24_1 1298 1320 1313 1313
query25 569 474 403 403
query26 1295 321 170 170
query27 2717 574 369 369
query28 4453 1993 1988 1988
query29 1101 618 516 516
query30 314 231 201 201
query31 1168 1093 1000 1000
query32 118 74 61 61
query33 538 336 266 266
query34 1184 1139 672 672
query35 790 794 692 692
query36 1411 1404 1236 1236
query37 155 108 98 98
query38 3209 3168 3047 3047
query39 925 913 900 900
query39_1 911 883 871 871
query40 223 131 106 106
query41 72 68 67 67
query42 102 110 105 105
query43 333 319 279 279
query44
query45 205 189 188 188
query46 1109 1167 773 773
query47 2383 2497 2299 2299
query48 412 413 311 311
query49 648 481 380 380
query50 982 357 265 265
query51 4326 4310 4292 4292
query52 91 91 77 77
query53 247 275 199 199
query54 288 232 226 226
query55 80 76 72 72
query56 261 243 241 241
query57 1450 1420 1348 1348
query58 259 231 222 222
query59 1628 1663 1437 1437
query60 320 265 239 239
query61 194 157 154 154
query62 689 646 594 594
query63 229 189 186 186
query64 2525 769 614 614
query65
query66 1803 471 346 346
query67 29818 29776 29560 29560
query68
query69 426 294 253 253
query70 973 907 949 907
query71 306 229 213 213
query72 2882 2679 2408 2408
query73 840 764 426 426
query74 5139 4979 4822 4822
query75 2666 2615 2245 2245
query76 2350 1165 775 775
query77 372 375 283 283
query78 12598 12527 11966 11966
query79 1442 1036 748 748
query80 848 479 396 396
query81 490 285 251 251
query82 819 154 134 134
query83 366 283 242 242
query84
query85 886 533 431 431
query86 406 298 290 290
query87 3414 3380 3232 3232
query88 3636 2724 2735 2724
query89 425 387 330 330
query90 1807 185 178 178
query91 179 160 136 136
query92 68 60 57 57
query93 1430 1478 857 857
query94 634 353 311 311
query95 683 392 354 354
query96 1048 784 346 346
query97 2729 2690 2555 2555
query98 214 213 222 213
query99 1150 1187 1060 1060
Total cold run time: 252561 ms
Total hot run time: 169813 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4604298058
<details>
<summary>TPC-H: <b>Total hot run time: 29058 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 14da24479173944ea5601dd5c550fa43e7079e0d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17718 3979 3983 3979
q2 q3 10764 1371 801 801
q4 4689 464 340 340
q5 7591 872 595 595
q6 182 167 134 134
q7 796 865 628 628
q8 9427 1570 1643 1570
q9 5893 4441 4478 4441
q10 6753 1814 1571 1571
q11 439 273 253 253
q12 626 422 288 288
q13 18105 3341 2729 2729
q14 267 252 243 243
q15 q16 825 766 709 709
q17 995 1012 850 850
q18 7185 5891 5652 5652
q19 1292 1305 1008 1008
q20 495 411 259 259
q21 6239 2773 2694 2694
q22 480 386 314 314
Total cold run time: 100761 ms
Total hot run time: 29058 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5052 4704 4757 4704
q2 q3 4836 5327 4621 4621
q4 2130 2214 1381 1381
q5 4736 4849 4619 4619
q6 235 178 136 136
q7 1875 1875 1577 1577
q8 2438 2053 2071 2053
q9 7800 7424 7307 7307
q10 4786 4685 4229 4229
q11 522 379 347 347
q12 719 734 521 521
q13 2969 3361 2797 2797
q14 270 289 249 249
q15 q16 672 694 608 608
q17 1272 1242 1237 1237
q18 7215 6700 6658 6658
q19 1132 1073 1123 1073
q20 2203 2218 1944 1944
q21 5240 4548 4393 4393
q22 506 446 419 419
Total cold run time: 56608 ms
Total hot run time: 50873 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4613010669
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4621051536
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4623937888
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4623939343
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4628996459
<details>
<summary>TPC-H: <b>Total hot run time: 29234 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit afba38054f35638e61af5da67ffb46085f0b55e7, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 18006 4047 4001 4001
q2 q3 10802 1362 800 800
q4 4684 473 354 354
q5 7536 870 608 608
q6 183 175 140 140
q7 797 846 627 627
q8 9384 1538 1616 1538
q9 5771 4500 4505 4500
q10 6782 1794 1534 1534
q11 437 281 252 252
q12 638 429 293 293
q13 18127 3394 2817 2817
q14 267 262 238 238
q15 q16 829 789 708 708
q17 980 880 997 880
q18 7103 6008 5733 5733
q19 1356 1340 1141 1141
q20 524 405 263 263
q21 5881 2554 2508 2508
q22 436 350 299 299
Total cold run time: 100523 ms
Total hot run time: 29234 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4330 4214 4243 4214
q2 q3 4551 4959 4360 4360
q4 2087 2197 1377 1377
q5 4433 4327 4319 4319
q6 230 180 129 129
q7 1722 1629 2093 1629
q8 2735 2246 2168 2168
q9 8140 8338 7996 7996
q10 4848 4763 4286 4286
q11 554 407 397 397
q12 741 777 546 546
q13 3378 3722 2965 2965
q14 292 315 270 270
q15 q16 722 751 645 645
q17 1341 1334 1311 1311
q18 8001 7384 7207 7207
q19 1199 1151 1149 1149
q20 2234 2206 1951 1951
q21 5250 4543 4450 4450
q22 538 466 410 410
Total cold run time: 57326 ms
Total hot run time: 51779 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4622186736
<details>
<summary>TPC-H: <b>Total hot run time: 29456 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa859dd138717a39731d64ec330321c62f5f4cc4, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17624 4169 4010 4010
q2 q3 10833 1369 849 849
q4 4690 482 346 346
q5 7603 894 582 582
q6 188 183 144 144
q7 773 866 645 645
q8 9389 1624 1616 1616
q9 5793 4467 4503 4467
q10 6807 1815 1551 1551
q11 433 277 254 254
q12 629 436 303 303
q13 18131 3354 2770 2770
q14 269 255 242 242
q15 q16 823 775 715 715
q17 1003 978 1065 978
q18 6959 5665 5686 5665
q19 1291 1280 1037 1037
q20 527 404 267 267
q21 6296 2811 2688 2688
q22 461 389 327 327
Total cold run time: 100522 ms
Total hot run time: 29456 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5195 4872 5026 4872
q2 q3 4881 5327 4663 4663
q4 2149 2234 1390 1390
q5 4862 4905 4746 4746
q6 232 188 140 140
q7 1838 1757 1667 1667
q8 2478 2175 2111 2111
q9 8000 7880 7440 7440
q10 4711 4676 4203 4203
q11 548 413 370 370
q12 732 730 536 536
q13 3007 3335 2771 2771
q14 267 277 257 257
q15 q16 677 698 622 622
q17 1294 1247 1257 1247
q18 7317 6908 6773 6773
q19 1138 1125 1124 1124
q20 2195 2224 1957 1957
q21 5328 4619 4555 4555
q22 522 462 409 409
Total cold run time: 57371 ms
Total hot run time: 51853 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4631874637
<details>
<summary>TPC-H: <b>Total hot run time: 28899 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90b0103ac670a6a7745e940394f4a283481be422, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17779 3995 3958 3958
q2 q3 10756 1420 797 797
q4 4686 471 341 341
q5 7633 872 590 590
q6 181 172 135 135
q7 765 849 650 650
q8 9400 1510 1488 1488
q9 6122 4486 4422 4422
q10 6684 1797 1549 1549
q11 452 271 258 258
q12 637 426 299 299
q13 18139 3311 2769 2769
q14 267 260 235 235
q15 q16 814 772 705 705
q17 964 988 911 911
q18 6970 5591 5470 5470
q19 1319 1279 1067 1067
q20 529 385 279 279
q21 6212 2744 2659 2659
q22 467 371 317 317
Total cold run time: 100776 ms
Total hot run time: 28899 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4847 4646 4597 4597
q2 q3 5085 5204 4632 4632
q4 2123 2219 1379 1379
q5 4873 4791 4708 4708
q6 239 179 126 126
q7 1919 1780 1585 1585
q8 2474 2195 2007 2007
q9 7302 7457 7414 7414
q10 4702 4658 4207 4207
q11 533 394 372 372
q12 723 742 532 532
q13 2929 3370 2828 2828
q14 270 286 246 246
q15 q16 673 701 626 626
q17 1282 1254 1262 1254
q18 7257 6817 6923 6817
q19 1109 1077 1079 1077
q20 2213 2213 1937 1937
q21 5269 4655 4502 4502
q22 532 453 429 429
Total cold run time: 56354 ms
Total hot run time: 51275 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4552786591
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3332813478
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,452 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked);
+ }
+ }
+
+ // =========================================================================
+ // Pull-up eligibility
+ // =========================================================================
+
+ /**
+ * Check if a named expression can be pulled up above TopN.
+ * Eligible: Alias with non-trivial child, not blocked, no NoneMovableFunction.
+ */
+ static boolean canPullUp(NamedExpression ne) {
+ if (!(ne instanceof Alias)) {
+ return false;
+ }
+ Expression child = ((Alias) ne).child();
+ if (child instanceof Slot || child instanceof Literal) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof NoneMovableFunction)) {
+ return false;
+ }
+ return true;
+ }
+
+ private static boolean isBlockingNode(Plan node) {
+ return node instanceof LogicalAggregate
+ || node instanceof LogicalWindow
+ || node instanceof LogicalRepeat;
+ }
+
+ private static Set<ExprId> buildOrderKeyExprIds(LogicalTopN<?> topN) {
+ Set<ExprId> orderKeyExprIds = new HashSet<>();
+ for (OrderKey orderKey : topN.getOrderKeys()) {
+ Expression keyExpr = orderKey.getExpr();
+ orderKeyExprIds.addAll(keyExpr.getInputSlotExprIds());
+ if (keyExpr instanceof NamedExpression) {
+ orderKeyExprIds.add(((NamedExpression) keyExpr).getExprId());
+ }
+ }
+ return orderKeyExprIds;
+ }
+
+ /**
+ * Deduplicate pull-up expressions so that each expression in a Project is only
+ * pulled up to the outermost TopN that collects it.
+ *
+ * <p>Since {@link CollectorContext#topNToPullUpInfo} is a {@link LinkedHashMap}
+ * and the Collector visits the plan top-down, iteration order is outer-to-inner.
+ * We keep the first occurrence of each (project-reference, exprId) pair and
+ * remove duplicates from inner TopNs.
+ */
+ private static void deduplicatePullUps(CollectorContext context) {
+ // Use IdentityHashMap because we need to distinguish Project nodes by object
+ // reference, not by content equality.
+ Map<LogicalProject<? extends Plan>, Set<ExprId>> handled = new IdentityHashMap<>();
+
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> toRemove = new ArrayList<>();
+ for (Map.Entry<LogicalProject<? extends Plan>, List<NamedExpression>> entry
+ : info.projectToPulledUpExprs.entrySet()) {
+ LogicalProject<? extends Plan> project = entry.getKey();
+ Set<ExprId> projectHandled = handled.computeIfAbsent(project, k -> new HashSet<>());
+ for (NamedExpression expr : entry.getValue()) {
+ if (projectHandled.contains(expr.getExprId())) {
+ toRemove.add(expr);
+ } else {
+ projectHandled.add(expr.getExprId());
+ }
+ }
+ }
+ for (NamedExpression expr : toRemove) {
+ info.allPulledUpExprs.remove(expr);
+ for (List<NamedExpression> list : info.projectToPulledUpExprs.values()) {
+ list.removeIf(e -> e == expr);
+ }
+ info.projectToPulledUpExprs.entrySet().removeIf(e -> e.getValue().isEmpty());
+ info.baseSlotsByExpr.remove(expr.getExprId());
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 2: Replacer (bottom-up)
+ // =========================================================================
+
+ static class Replacer extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, CollectorContext context) {
+ LogicalProject<? extends Plan> rewritten = (LogicalProject<? extends Plan>) visit(project, context);
+
+ // Collect ALL pulled-up expressions across ALL PullUpInfos for this
+ // project. After dedup, each expression belongs to exactly one TopN
+ // (the outermost one that can pull it up). The project needs to be
+ // simplified by removing all of them, exposing their base slots once.
+ List<NamedExpression> allPulledUpExprs = collectAllPulledUpExprs(context, rewritten);
+ if (allPulledUpExprs.isEmpty() && rewritten != project
+ && rewritten.getProjects().equals(project.getProjects())) {
+ allPulledUpExprs = collectAllPulledUpExprs(context, project);
+ }
+ if (allPulledUpExprs.isEmpty()) {
+ return rewritten;
+ }
+ return simplifyProject(rewritten, allPulledUpExprs, context);
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ LogicalTopN rewritten = (LogicalTopN) visit(topN, context);
+ PullUpInfo info = context.getPullUpInfo(rewritten);
+ if (info == null && rewritten != topN) {
+ info = context.getPullUpInfo(topN);
+ }
+ if (info == null || info.allPulledUpExprs.isEmpty()) {
+ return rewritten;
+ }
+ return addUpperProject(rewritten, info);
+ }
+ }
+
+ /**
+ * Collect all pulled-up expressions across all PullUpInfos for a project.
+ * After dedup each expression belongs to exactly one TopN, but the project
+ * must be simplified by removing all of them at once.
+ */
+ private static List<NamedExpression> collectAllPulledUpExprs(
+ CollectorContext context, LogicalProject<?> project) {
+ List<NamedExpression> result = new ArrayList<>();
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> exprs = info.projectToPulledUpExprs.get(project);
+ if (exprs != null) {
+ result.addAll(exprs);
+ }
+ }
+ return result;
+ }
+
+ /** Remove pulled-up expressions from project and add their base input slots. */
+ private static LogicalProject<? extends Plan> simplifyProject(
+ LogicalProject<? extends Plan> project,
+ List<NamedExpression> pulledUpExprs,
+ CollectorContext context) {
+ if (pulledUpExprs.isEmpty()) {
+ return project;
+ }
+
+ Set<ExprId> pulledUpExprIds = new HashSet<>();
+ for (NamedExpression ne : pulledUpExprs) {
+ pulledUpExprIds.add(ne.getExprId());
+ }
+
+ List<NamedExpression> simplified = new ArrayList<>();
+ Set<ExprId> existingExprIds = new HashSet<>();
+ for (NamedExpression ne : project.getProjects()) {
+ if (!pulledUpExprIds.contains(ne.getExprId())) {
+ simplified.add(ne);
+ existingExprIds.add(ne.getExprId());
+ }
+ }
+
+ for (NamedExpression pulledUpExpr : pulledUpExprs) {
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<Slot> baseSlots = info.baseSlotsByExpr.get(pulledUpExpr.getExprId());
+ if (baseSlots != null) {
+ for (Slot baseSlot : baseSlots) {
+ if (!existingExprIds.contains(baseSlot.getExprId())) {
+ simplified.add(baseSlot);
+ existingExprIds.add(baseSlot.getExprId());
+ }
+ }
+ break; // found, no need to check other PullUpInfos
+ }
+ }
+ }
+
+ if (simplified.equals(project.getProjects())) {
+ return project;
+ }
+ return (LogicalProject<? extends Plan>) project.withProjects(simplified);
+ }
+
+ /** Create a new Project above the TopN that restores pulled-up expressions. */
+ private static LogicalProject<Plan> addUpperProject(LogicalTopN topN, PullUpInfo info) {
+ Map<ExprId, NamedExpression> pulledUpBySlotExprId = new HashMap<>();
+ for (NamedExpression e : info.allPulledUpExprs) {
+ pulledUpBySlotExprId.put(e.toSlot().getExprId(), e);
+ }
+
+ // Use the current (possibly rewritten) TopN's output so that slots
+ // whose expressions were deduplicated to an outer TopN reference
+ // the correct post-simplification ExprIds instead of stale ones.
+ List<Slot> currentOutput = topN.getOutput();
+ List<NamedExpression> upperOutput = new ArrayList<>();
+ for (int i = 0; i < info.originalTopNOutput.size(); i++) {
+ Slot origSlot = info.originalTopNOutput.get(i);
+ NamedExpression pulledUpExpr = pulledUpBySlotExprId.get(origSlot.getExprId());
+ if (pulledUpExpr != null) {
+ upperOutput.add(pulledUpExpr);
+ } else if (i < currentOutput.size()) {
+ // Slot was not pulled up: use the current slot at the same
+ // position so the ExprId matches the rewritten child subtree.
+ upperOutput.add(currentOutput.get(i));
Review Comment:
This ordinal lookup is not safe after `simplifyProject()` removes pulled expressions and appends their base slots. For example, with a lower project output `[id, x = a + 1, c]` under a TopN where only `x` is pulled up, the simplified child output becomes `[id, c, a]`. When rebuilding the upper project for the original output, `c` is not in `pulledUpBySlotExprId`, so this branch uses `currentOutput.get(2)`, which is `a`, producing `[id, x, a]` instead of `[id, x, c]`. Please map non-pulled original slots to the rewritten TopN output by ExprId (or otherwise preserve a slot mapping through simplification) rather than by ordinal, and add a regression/unit test where a pulled expression is followed by a non-pulled output column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4617819492
<details>
<summary>TPC-H: <b>Total hot run time: 29044 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa8526456fd8a1e1a2f7800f79ffb445388e4953, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17766 4091 4149 4091
q2 q3 10790 1416 808 808
q4 4714 476 357 357
q5 7712 851 602 602
q6 203 181 137 137
q7 827 826 647 647
q8 10131 1557 1635 1557
q9 6791 4529 4548 4529
q10 6849 1812 1525 1525
q11 448 273 249 249
q12 650 422 296 296
q13 18155 3403 2773 2773
q14 275 264 241 241
q15 q16 824 772 704 704
q17 917 925 947 925
q18 6782 5741 5643 5643
q19 1299 1245 987 987
q20 501 414 269 269
q21 5956 2640 2409 2409
q22 428 356 295 295
Total cold run time: 102018 ms
Total hot run time: 29044 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4454 4427 4314 4314
q2 q3 4501 4903 4338 4338
q4 2106 2228 1400 1400
q5 4452 4334 4353 4334
q6 227 175 131 131
q7 1765 2163 1686 1686
q8 2556 2249 2229 2229
q9 8081 8062 8046 8046
q10 4789 4768 4271 4271
q11 587 600 404 404
q12 748 750 539 539
q13 3325 3550 3034 3034
q14 307 302 294 294
q15 q16 743 748 642 642
q17 1382 1356 1373 1356
q18 7988 7420 6903 6903
q19 1099 1117 1096 1096
q20 2243 2218 1988 1988
q21 5290 4655 4497 4497
q22 511 452 411 411
Total cold run time: 57154 ms
Total hot run time: 51913 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3351230397
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,642 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+ final Map<ExprId, NamedExpression> passThroughExprByDeduplicatedExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+
+ void addPassThroughExprForDeduplicatedExpr(NamedExpression expr) {
+ passThroughExprByDeduplicatedExpr.put(expr.getExprId(), expr);
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ final Map<Slot, Expression> pullUpExprReplaceMap = new LinkedHashMap<>();
+ /**
+ * When collectFromNode encounters a nested TopN, it saves the current
+ * blockedExprIds (accumulated from outer nodes) so that visitLogicalTopN
+ * for the inner TopN can merge them into its fresh blocked set.
+ */
+ final Map<LogicalTopN, Set<ExprId>> outerBlockedByTopN = new IdentityHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ void addPullUpExprReplace(NamedExpression expr) {
+ if (expr instanceof Alias) {
+ pullUpExprReplaceMap.putIfAbsent(expr.toSlot(), expr.child(0));
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ // If this is a nested TopN, merge in the outer blocked set that was
+ // saved by collectFromNode when it encountered this TopN. This
+ // ensures that slots consumed by outer operators (e.g. join
+ // conditions above this TopN) also block pull-up from projects
+ // under this TopN.
+ Set<ExprId> outerBlocked = context.outerBlockedByTopN.remove(topN);
+ if (outerBlocked != null) {
+ blockedExprIds.addAll(outerBlocked);
+ }
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds, context);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ for (NamedExpression expr : info.allPulledUpExprs) {
+ context.addPullUpExprReplace(expr);
+ }
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds,
+ CollectorContext context) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds, context);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // Save the current blockedExprIds (accumulated from outer nodes
+ // such as outer TopN + intermediate Joins) so that the inner
+ // TopN's own visitLogicalTopN can merge them into its fresh
+ // blocked set. Without this, outer join condition slots would
+ // not block pull-up from projects under the inner TopN.
+ context.outerBlockedByTopN.put(inner, new HashSet<>(blockedExprIds));
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked, context);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked, context);
Review Comment:
This traversal also walks through null-generating joins, but the blocked set only contains slots referenced by the join expressions. That lets the rule pull expressions from the nullable side of an outer join above the join, which is not semantics-preserving for non-null-preserving expressions. For example, `TopN(order by l.id) -> LEFT JOIN(left=l, right=Project(ifnull(r.b, 0) AS x, r.id))` currently can collect `x` because the join condition references `r.id` but not `x`. Before the rewrite, unmatched left rows get `x = NULL` from join null-extension of the right tuple; after simplification the upper project computes `ifnull(NULL, 0)` and returns `0`. Please treat null-generating join sides as boundaries, or at least block the outputs from those sides, and add a negative test for a nullable-side expression such as `ifnull`/`coalesce`.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/materialize/LazyMaterializeTopN.java:
##########
@@ -85,36 +79,82 @@ public Plan visitPhysicalTopN(PhysicalTopN topN, CascadesContext ctx) {
}
}
- private Plan computeTopN(PhysicalTopN topN, CascadesContext ctx) {
+ private Plan computeTopN(PhysicalTopN<? extends Plan> topN, CascadesContext ctx) {
if (hasMaterialized) {
return topN;
}
if (SessionVariable.getTopNLazyMaterializationThreshold() < topN.getLimit()) {
return topN;
}
- /*
- topn(output=[x] orderkey=[b])
- ->project(a as x)
- ->T(a, b)
- 'x' can be lazy materialized.
- materializeMap: x->(T, a)
- */
+ try {
+ List<Slot> userVisibleOutput = ImmutableList.copyOf(topN.getOutput());
+ List<Slot> effectiveOutput = ImmutableList.copyOf(topN.getOutput());
+ Plan result = doComputeTopN(topN, ctx, effectiveOutput);
+ if (result == topN) {
+ return topN;
+ }
+ result = new PhysicalProject(ImmutableList.copyOf(userVisibleOutput), null, result);
+ return result;
+ } catch (RuntimeException e) {
+ LOG.warn("lazy materialize topn failed for plan: {}", topN.shapeInfo(), e);
+ return topN;
+ }
+ }
+
+ private LazySlotPruning createLazySlotPruning() {
+ return new LazySlotPruning() {
+ @Override
+ protected boolean shouldPruneChild(Plan child, Context context) {
+ return true;
+ }
Review Comment:
Overriding `shouldPruneChild()` to always return true makes `LazySlotPruning` recurse into every child branch, not only the branch that contains `context.lazySlots`. In a join-shaped TopN where only one relation has lazy slots, the visitor also reaches the sibling scan; `visitPhysicalOlapScan()`/`visitPhysicalFileScan()` then sees that the sibling output does not contain the lazy slots and throws `Lazy materialize fault`. `computeTopN()` catches that `RuntimeException` and returns the original TopN, so join cases that were otherwise eligible silently lose lazy materialization. Please keep the original output-containment guard for unrelated branches and address stale logical properties in a narrower way, and add a test with TopN above a join where only one side is lazily materialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4622574349
# FE Regression Coverage Report
Increment line coverage ` 83.53% (416/498)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_fa859dd138717a39731d64ec330321c62f5f4cc4_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_fa859dd138717a39731d64ec330321c62f5f4cc4_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4592715234
<details>
<summary>TPC-DS: <b>Total hot run time: 169411 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cefd39843930a5392f14a48b3c47d033bb40e59d, data reload: false
query5 4321 637 514 514
query6 352 231 207 207
query7 4316 567 310 310
query8 324 227 216 216
query9 8819 4000 3979 3979
query10 443 356 300 300
query11 5838 2339 2143 2143
query12 187 132 136 132
query13 1296 613 448 448
query14 6108 5447 5149 5149
query14_1 4442 4443 4401 4401
query15 216 201 181 181
query16 1020 471 419 419
query17
query18 2529 498 358 358
query19 213 206 185 185
query20 144 138 130 130
query21 212 139 124 124
query22 13597 13588 13299 13299
query23 17224 16429 16386 16386
query23_1 16213 16391 16289 16289
query24 8195 1780 1341 1341
query24_1 1342 1321 1276 1276
query25 633 490 437 437
query26 1361 340 177 177
query27 2690 589 351 351
query28 4536 2033 1989 1989
query29 1080 665 554 554
query30 314 235 205 205
query31 1124 1072 973 973
query32 89 80 76 76
query33 557 366 316 316
query34 1176 1163 670 670
query35 793 818 700 700
query36 1420 1441 1247 1247
query37 157 105 92 92
query38 3213 3165 3119 3119
query39 945 970 920 920
query39_1 903 889 914 889
query40 242 154 131 131
query41 71 68 69 68
query42 118 118 115 115
query43 334 330 300 300
query44
query45 235 214 196 196
query46 1091 1223 767 767
query47 2437 2437 2311 2311
query48 426 427 295 295
query49 649 540 413 413
query50 1019 363 268 268
query51 4311 4372 4266 4266
query52 110 112 99 99
query53 266 282 205 205
query54 337 304 285 285
query55 109 92 89 89
query56 339 331 322 322
query57 1485 1421 1391 1391
query58
query59 1565 1643 1434 1434
query60 354 364 328 328
query61 182 186 187 186
query62 704 651 600 600
query63 256 222 227 222
query64 2469 871 632 632
query65
query66 1721 477 359 359
query67 29926 29654 29485 29485
query68
query69 474 329 287 287
query70 1066 999 999 999
query71 336 278 266 266
query72 3005 2745 2490 2490
query73 868 758 421 421
query74 5133 4940 4808 4808
query75 2677 2638 2264 2264
query76 2389 1142 784 784
query77 397 406 339 339
query78 12364 12426 11751 11751
query79 1470 1026 790 790
query80 1336 516 464 464
query81 521 283 240 240
query82 980 153 119 119
query83
query84
query85 935 529 455 455
query86 466 336 342 336
query87 3485 3365 3225 3225
query88 3642 2781 2754 2754
query89 441 390 342 342
query90 1907 187 180 180
query91 179 160 141 141
query92 81 75 74 74
query93 1623 1403 850 850
query94 740 357 311 311
query95 682 477 344 344
query96 1015 765 360 360
query97 2754 2740 2629 2629
query98 234 232 232 232
query99 1176 1182 1025 1025
Total cold run time: 254570 ms
Total hot run time: 169411 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4565803791
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3341962899
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,622 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+ final Map<ExprId, NamedExpression> passThroughExprByDeduplicatedExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+
+ void addPassThroughExprForDeduplicatedExpr(NamedExpression expr) {
+ passThroughExprByDeduplicatedExpr.put(expr.getExprId(), expr);
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ final Map<Slot, Expression> pullUpExprReplaceMap = new LinkedHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ void addPullUpExprReplace(NamedExpression expr) {
+ if (expr instanceof Alias) {
+ pullUpExprReplaceMap.putIfAbsent(expr.toSlot(), expr.child(0));
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ for (NamedExpression expr : info.allPulledUpExprs) {
+ context.addPullUpExprReplace(expr);
+ }
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked);
+ }
+ }
+
+ // =========================================================================
+ // Pull-up eligibility
+ // =========================================================================
+
+ /**
+ * Check if a named expression can be pulled up above TopN.
+ * Eligible: Alias with non-trivial child, not blocked, no NoneMovableFunction.
+ */
+ static boolean canPullUp(NamedExpression ne) {
+ if (!(ne instanceof Alias)) {
+ return false;
+ }
+ Expression child = ((Alias) ne).child();
+ if (child instanceof Slot || child instanceof Literal) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof NoneMovableFunction)) {
Review Comment:
`canPullUp()` does not reject volatile/unique expressions. A plan shaped like `TopN(order by id) -> Project(random() AS r, id) -> Scan` satisfies the current checks because `random()` is an `Alias` child that is neither a `Slot`, `Literal`, `NoneMovableFunction`, `Score`, nor variant slot, so the rule moves it above TopN. Other Nereids movement rules explicitly guard `containsVolatileExpression()` because `UniqueFunction` expressions carry per-occurrence identity and should not be relocated across operators. Please add a volatile-expression check here and a negative unit test so functions such as `random()`/`random_bytes()` stay in their original evaluation position.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4602926902
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4600038088
<details>
<summary>TPC-H: <b>Total hot run time: 29335 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0862cc18bf424da86e16acff60787158a0febcd, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17669 4222 4172 4172
q2 q3 10759 1441 829 829
q4 4681 468 341 341
q5 7536 890 588 588
q6 182 176 143 143
q7 798 830 658 658
q8 9343 1570 1608 1570
q9 6277 4498 4447 4447
q10 6823 1810 1530 1530
q11 437 278 255 255
q12 646 421 289 289
q13 18159 3480 2770 2770
q14 272 255 238 238
q15 q16 812 782 719 719
q17 998 944 921 921
q18 7043 5709 5699 5699
q19 1316 1287 993 993
q20 506 408 270 270
q21 6014 2848 2578 2578
q22 460 386 325 325
Total cold run time: 100731 ms
Total hot run time: 29335 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5114 5045 4967 4967
q2 q3 5190 5271 4646 4646
q4 2179 2264 1412 1412
q5 4901 4905 4896 4896
q6 254 187 131 131
q7 1853 1741 1626 1626
q8 2576 2226 2084 2084
q9 7492 7472 7434 7434
q10 4747 4697 4200 4200
q11 564 423 382 382
q12 743 750 548 548
q13 2974 3494 2755 2755
q14 278 281 255 255
q15 q16 680 698 626 626
q17 1293 1276 1264 1264
q18 7388 7002 6969 6969
q19 1180 1098 1128 1098
q20 2226 2225 1967 1967
q21 5386 4672 4559 4559
q22 529 464 408 408
Total cold run time: 57547 ms
Total hot run time: 52227 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4602422615
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4555885127
# FE Regression Coverage Report
Increment line coverage ` 75.15% (127/169)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_4897f0cfc299fd90b64a17d6658e5891a6bbc55b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_4897f0cfc299fd90b64a17d6658e5891a6bbc55b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4629060701
<details>
<summary>TPC-DS: <b>Total hot run time: 170084 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit afba38054f35638e61af5da67ffb46085f0b55e7, data reload: false
query5 4322 639 480 480
query6 444 219 184 184
query7 4828 526 308 308
query8 383 227 199 199
query9 8767 4046 4043 4043
query10 466 327 270 270
query11 5896 2353 2135 2135
query12 155 102 97 97
query13 1250 624 426 426
query14 6439 5451 5090 5090
query14_1 4409 4411 4410 4410
query15 213 200 176 176
query16 1002 466 462 462
query17 1144 744 606 606
query18 2471 494 360 360
query19 211 189 150 150
query20 114 108 115 108
query21 212 138 120 120
query22 13693 13571 13472 13472
query23 17371 16506 16308 16308
query23_1 16355 16395 16327 16327
query24 7559 1782 1319 1319
query24_1 1336 1299 1279 1279
query25 601 481 412 412
query26 1319 334 177 177
query27 2650 580 335 335
query28 4440 2038 2064 2038
query29 1091 636 515 515
query30 306 246 204 204
query31 1136 1086 959 959
query32 113 63 61 61
query33 552 340 293 293
query34 1190 1139 633 633
query35 758 778 697 697
query36 1367 1364 1212 1212
query37 151 103 89 89
query38 3203 3147 3051 3051
query39 938 911 889 889
query39_1 875 879 880 879
query40 221 125 102 102
query41 64 66 61 61
query42 94 95 91 91
query43 317 320 282 282
query44
query45 199 187 179 179
query46 1136 1200 729 729
query47 2360 2352 2317 2317
query48 412 403 293 293
query49 633 479 350 350
query50 985 338 257 257
query51 4349 4342 4314 4314
query52 90 89 78 78
query53 247 265 192 192
query54 268 223 206 206
query55 80 76 70 70
query56 247 229 212 212
query57 1433 1404 1332 1332
query58 246 221 207 207
query59 1535 1604 1421 1421
query60 290 243 242 242
query61 166 163 160 160
query62 697 652 589 589
query63 231 181 187 181
query64 2603 800 614 614
query65
query66 1797 463 338 338
query67 29709 29699 29629 29629
query68
query69 409 308 268 268
query70 976 979 979 979
query71 319 223 209 209
query72 3014 2723 2463 2463
query73 824 770 423 423
query74 5116 4948 4766 4766
query75 2675 2618 2250 2250
query76 2318 1139 817 817
query77 360 377 289 289
query78 12482 12446 11865 11865
query79 1436 1030 726 726
query80 596 459 394 394
query81 452 281 246 246
query82 564 160 124 124
query83 359 271 246 246
query84
query85 878 538 464 464
query86 366 302 271 271
query87 3410 3350 3209 3209
query88 3657 2764 2737 2737
query89 429 380 342 342
query90 1946 177 178 177
query91 183 165 143 143
query92 62 89 59 59
query93 1428 1446 901 901
query94 556 359 317 317
query95 687 447 346 346
query96 1116 783 364 364
query97 2731 2713 2581 2581
query98 218 209 200 200
query99 1156 1195 1064 1064
Total cold run time: 251363 ms
Total hot run time: 170084 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3360173865
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/materialize/MaterializeProbeVisitor.java:
##########
@@ -149,7 +158,7 @@ public Optional<MaterializeSource> visitPhysicalOlapScan(PhysicalOlapScan scan,
if (KeysType.AGG_KEYS.equals(table.getKeysType())) {
return Optional.empty();
}
Review Comment:
Dropping the `scan.getOperativeSlots()` guard makes scan-level required columns eligible for lazy materialization. `OperativeColumnDerive` still marks slots needed by scan virtual columns and other scan-side work as operative, but `collectProjectExprInputSlots()` only reconstructs project/key/non-original requirements and does not add those operative slots. For a scan with a virtual column whose base slot is also a TopN output, this probe can now return a `MaterializeSource`; `LazySlotPruning` then removes that slot from `PhysicalLazyMaterializeOlapScan.computeOutput()`, while the underlying scan still needs it to compute the operative/virtual expression. Please keep the existing operative-slot rejection in addition to the new `requiredMaterializedSlots` check, and apply the same rule to the catalog/TVF relation branches that also dropped `getOperativeSlots()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #63736:
URL: https://github.com/apache/doris/pull/63736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3325577962
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,486 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalExcept;
+import org.apache.doris.nereids.trees.plans.logical.LogicalIntersect;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Pass 2: Replace/restructure
Review Comment:
`deduplicatePullUps()` is never called, and `getPullUpInfoForProject()` returns only the first `PullUpInfo` that references a project. In a nested TopN shape like the new `testDeduplicatePullUpToOutermostTopN` scenario, the outer TopN records only `y` while the inner TopN records `x` and `y`; when the bottom project is rewritten, it is simplified with the outer info only, so `x` remains computed below the inner TopN even though an upper project is also added for it. That means the inner-only expression is not actually pulled out of the TopN input, so the intended lazy-materialization benefit is lost and the plan computes the expression twice. Please either call the deduplication pass before replacement and/or make project simplification account for each TopN's own pulled expressions.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,486 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalExcept;
+import org.apache.doris.nereids.trees.plans.logical.LogicalIntersect;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalUnion;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ PullUpInfo getPullUpInfoForProject(LogicalProject<? extends Plan> project) {
+ for (PullUpInfo info : topNToPullUpInfo.values()) {
+ if (info.projectToPulledUpExprs.containsKey(project)) {
+ return info;
+ }
+ }
+ return null;
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // For set operators, walk into each child.
+ // blockedExprIds must be mapped from the set operator's output ExprIds
+ // to each child's output ExprIds via regularChildrenOutputs.
+ //
+ // Union ALL is transparent: it only concatenates rows and does not
Review Comment:
This traversal makes the current TopN collect expressions from below a set operator. That is not semantics-preserving for UNION ALL unless every child computes exactly the same expression. For example, `select x, id from (select a + 1 as x, id from t1 union all select a + 2 as x, id from t2) u order by id limit 10`: `x` is not an order key, so both child `Project` expressions are collected and simplified to expose `a`. The single project added above the TopN cannot represent the branch-specific `a + 1` versus `a + 2` semantics (and may also reference only one child expression id), so rows from one branch can get the wrong value or the union inputs can stop matching their declared child outputs. Please treat set operations as a boundary for the current TopN; it is fine to let the normal visitor continue into their children so nested TopNs are handled independently.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/materialize/LazyMaterializeTopN.java:
##########
@@ -154,47 +159,28 @@ private Plan computeTopN(PhysicalTopN topN, CascadesContext ctx) {
tvfRelation.getFunction().getName() + ".global_row_id", false, Integer.MAX_VALUE);
SlotReference rowIdSlot = SlotReference.fromColumn(threadStatementContext.getNextExprId(),
tvfRelation.getFunction().getTable(), rowIdCol, ImmutableList.of());
- result = result.accept(new LazySlotPruning(),
- new LazySlotPruning.Context((PhysicalTVFRelation) relation,
- rowIdSlot, relationToLazySlotMap.get(relation)));
+ result = result.accept(createLazySlotPruning(), new LazySlotPruning.Context(
+ (PhysicalTVFRelation) relation,
+ rowIdSlot, relationToLazySlotMap.get(relation)));
relationToRowId.put(tvfRelation, rowIdSlot);
rowIdSet.add(rowIdSlot);
} else {
- // should not reach here.
throw new RuntimeException("LazyMaterializeTopN not support this relation." + relation);
}
}
- // materialize.child.output requires
- // rowId only appears once.
- // that is [a, rowId1, b rowId1] is not acceptable
List<SlotReference> materializeInput = moveRowIdsToTail(result.getOutput(), rowIdSet);
if (materializeInput == null) {
- /*
- topn
- -->any
- =>
- project
- -->materialize
- -->topn
- -->any
- */
- result = new PhysicalLazyMaterialize(result, result.getOutput(),
+ List<Slot> correctInput = new ArrayList<>(materializedSlots);
+ for (SlotReference rowId : relationToRowId.values()) {
+ correctInput.add(rowId);
Review Comment:
When `moveRowIdsToTail(result.getOutput(), rowIdSet)` returns `null`, the child output is already in a valid `[materialized slots..., row ids...]` layout. Rebuilding `correctInput` from `materializedSlots` plus `relationToRowId.values()` can change the row-id order for multi-relation lazy materialization because the row ids are stored in hash-based maps. `PhysicalLazyMaterialize` uses this list to derive the row-id/fetch relation order, so a plan with lazy columns from two joined relations can bind the fetch expression order differently from the actual child tuple layout. Please preserve `result.getOutput()` in this branch instead of reconstructing the input list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4591506791
# FE UT Coverage Report
Increment line coverage ` 76.55% (235/307)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_cefd39843930a5392f14a48b3c47d033bb40e59d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_cefd39843930a5392f14a48b3c47d033bb40e59d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4592684482
# FE Regression Coverage Report
Increment line coverage ` 52.46% (245/467)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_cefd39843930a5392f14a48b3c47d033bb40e59d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_cefd39843930a5392f14a48b3c47d033bb40e59d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4604481603
# FE Regression Coverage Report
Increment line coverage ` 77.53% (352/454)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_14da24479173944ea5601dd5c550fa43e7079e0d_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_14da24479173944ea5601dd5c550fa43e7079e0d_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4613106485
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4617675617
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4562717268
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4555396763
<details>
<summary>TPC-H: <b>Total hot run time: 32544 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4897f0cfc299fd90b64a17d6658e5891a6bbc55b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17656 4380 4235 4235
q2 q3 10791 1465 811 811
q4 4692 481 346 346
q5 7606 2333 2216 2216
q6 239 184 141 141
q7 974 811 636 636
q8 9396 1778 1792 1778
q9 5560 5082 5052 5052
q10 6424 2199 1905 1905
q11 445 276 256 256
q12 698 437 312 312
q13 18165 3446 2782 2782
q14 273 261 244 244
q15 q16 831 775 709 709
q17 1023 1031 1048 1031
q18 7032 5897 5483 5483
q19 1182 1495 1114 1114
q20 595 451 274 274
q21 6049 3012 2891 2891
q22 454 387 328 328
Total cold run time: 100085 ms
Total hot run time: 32544 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4974 4889 5177 4889
q2 q3 5013 5253 4648 4648
q4 2284 2309 1487 1487
q5 5106 4828 4736 4736
q6 249 189 135 135
q7 1914 1774 1642 1642
q8 2621 2422 2303 2303
q9 7889 7506 7453 7453
q10 4762 4722 4207 4207
q11 576 431 387 387
q12 750 755 550 550
q13 3063 3419 2815 2815
q14 271 276 260 260
q15 q16 703 704 617 617
q17 1334 1307 1300 1300
q18 7331 7142 6830 6830
q19 1133 1147 1151 1147
q20 2234 2230 1963 1963
q21 5432 4820 4730 4730
q22 528 475 422 422
Total cold run time: 58167 ms
Total hot run time: 52521 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4565743458
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4598457092
# FE UT Coverage Report
Increment line coverage ` 78.69% (277/352)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_f0862cc18bf424da86e16acff60787158a0febcd/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_f0862cc18bf424da86e16acff60787158a0febcd/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4598725548
# FE Regression Coverage Report
Increment line coverage ` 59.96% (268/447)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_f0862cc18bf424da86e16acff60787158a0febcd_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_f0862cc18bf424da86e16acff60787158a0febcd_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4602421743
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3341283747
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PullUpProjectExprUnderTopN.java:
##########
@@ -0,0 +1,624 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ExprId;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Score;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalCTEProducer;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRelation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSetOperation;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.trees.plans.logical.LogicalWindow;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.SessionVariable;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * Pull up non-trivial expressions from Projects below TopN to above TopN,
+ * exposing their input base columns as lazy materialization candidates.
+ *
+ * <p>Two-pass CustomRewriter:
+ * <ol>
+ * <li><b>Collector (top-down)</b>: walk the plan tree, find qualifying TopNs,
+ * walk into their descendants to find Projects with pull-able expressions.
+ * Any operator that references a slot blocks pulling up expressions that
+ * output that slot past it. Boundary nodes (Aggregate, Window, Repeat,
+ * Relation, CTEProducer) stop the walk.
+ * Set operators are treated as blockers for the current TopN but their
+ * children are still traversed so nested TopNs inside them are visited.</li>
+ * <li><b>Replacer (bottom-up)</b>: simplify found Projects and add upper
+ * Projects above TopN to restore pulled-up expressions.</li>
+ * </ol>
+ */
+public class PullUpProjectExprUnderTopN implements CustomRewriter {
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ ConnectContext ctx = jobContext.getCascadesContext()
+ .getStatementContext().getConnectContext();
+ if (ctx != null && !ctx.getSessionVariable().enableTopnExprPullup) {
+ return plan;
+ }
+
+ // Pass 1: Collect pull-up info
+ CollectorContext collectorCtx = new CollectorContext();
+ plan.accept(new Collector(), collectorCtx);
+
+ if (collectorCtx.topNToPullUpInfo.isEmpty()) {
+ return plan;
+ }
+
+ // Deduplicate: when nested TopNs both try to pull up the same expression
+ // from the same Project, keep it only in the outermost TopN.
+ deduplicatePullUps(collectorCtx);
+
+ // Pass 2: Replace/restructure
+ return plan.accept(new Replacer(), collectorCtx);
+ }
+
+ // =========================================================================
+ // Data structures
+ // =========================================================================
+
+ /** Info collected per TopN about which expressions to pull up from which Projects. */
+ static class PullUpInfo {
+ final LogicalTopN topN;
+ final List<Slot> originalTopNOutput;
+ final List<NamedExpression> allPulledUpExprs = new ArrayList<>();
+ final Map<LogicalProject<? extends Plan>, List<NamedExpression>> projectToPulledUpExprs
+ = new LinkedHashMap<>();
+ final Map<ExprId, List<Slot>> baseSlotsByExpr = new HashMap<>();
+ final Map<ExprId, NamedExpression> passThroughExprByDeduplicatedExpr = new HashMap<>();
+
+ PullUpInfo(LogicalTopN topN) {
+ this.topN = topN;
+ this.originalTopNOutput = ImmutableList.copyOf(topN.getOutput());
+ }
+
+ void addPulledUpExpr(LogicalProject<? extends Plan> project, NamedExpression expr) {
+ allPulledUpExprs.add(expr);
+ projectToPulledUpExprs.computeIfAbsent(project, k -> new ArrayList<>()).add(expr);
+ baseSlotsByExpr.put(expr.getExprId(), ImmutableList.copyOf(expr.getInputSlots()));
+ }
+
+ void addPassThroughExprForDeduplicatedExpr(NamedExpression expr) {
+ passThroughExprByDeduplicatedExpr.put(expr.getExprId(), expr);
+ }
+ }
+
+ /** Context shared between collector and replacer passes. */
+ static class CollectorContext {
+ final Map<LogicalTopN, PullUpInfo> topNToPullUpInfo = new LinkedHashMap<>();
+ final Map<Slot, Expression> pullUpExprReplaceMap = new LinkedHashMap<>();
+ int cteProducerDepth = 0;
+
+ boolean hasPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.containsKey(topN);
+ }
+
+ PullUpInfo getPullUpInfo(LogicalTopN topN) {
+ return topNToPullUpInfo.get(topN);
+ }
+
+ void addPullUpExprReplace(NamedExpression expr) {
+ if (expr instanceof Alias) {
+ pullUpExprReplaceMap.putIfAbsent(expr.toSlot(), expr.child(0));
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 1: Collector (top-down)
+ // =========================================================================
+
+ private static boolean qualifiesForLazyMatThreshold(LogicalTopN topN) {
+ long limit = topN.getLimit();
+ if (limit <= 0) {
+ return false;
+ }
+ long threshold = SessionVariable.getTopNLazyMaterializationThreshold();
+ return threshold >= limit;
+ }
+
+ static class Collector extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalCTEProducer(
+ LogicalCTEProducer<? extends Plan> cteProducer, CollectorContext context) {
+ context.cteProducerDepth++;
+ try {
+ return visit(cteProducer, context);
+ } finally {
+ context.cteProducerDepth--;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ if (context.cteProducerDepth > 0
+ || !qualifiesForLazyMatThreshold(topN)) {
+ return visit(topN, context);
+ }
+ PullUpInfo info = new PullUpInfo(topN);
+ // Seed blockedExprIds with this TopN's order key ExprIds so that
+ // expressions used by order keys are not pulled up past this TopN.
+ Set<ExprId> blockedExprIds = buildOrderKeyExprIds(topN);
+ collectFromNode((Plan) topN.child(0), info, blockedExprIds);
+ if (!info.allPulledUpExprs.isEmpty()) {
+ for (NamedExpression expr : info.allPulledUpExprs) {
+ context.addPullUpExprReplace(expr);
+ }
+ context.topNToPullUpInfo.put(topN, info);
+ }
+ return visit(topN, context);
+ }
+ }
+
+ /**
+ * Recursively walk down from a TopN's child to find Projects with pull-able expressions.
+ *
+ * <p>{@code blockedExprIds} contains ExprIds of slots that are referenced by operators
+ * along the path from the TopN to the current node. An expression whose output ExprId
+ * is in this set cannot be pulled up past the operators that reference it.
+ */
+ private static void collectFromNode(Plan node, PullUpInfo info, Set<ExprId> blockedExprIds) {
+ if (node instanceof LogicalProject) {
+ LogicalProject<? extends Plan> project = (LogicalProject<? extends Plan>) node;
+ for (NamedExpression ne : project.getProjects()) {
+ if (canPullUp(ne) && !blockedExprIds.contains(ne.getExprId())) {
+ info.addPulledUpExpr(project, ne);
+ }
+ }
+ // Continue into the project's child. Chained projects are all visited.
+ collectFromNode((Plan) project.child(0), info, blockedExprIds);
+ return;
+ }
+
+ if (node instanceof LogicalTopN) {
+ LogicalTopN inner = (LogicalTopN) node;
+ // TopN preserves all input columns, so it doesn't block by itself.
+ // However, its order keys consume slots, so add them to blocked set.
+ // Do NOT reset blockedExprIds — intermediate operators between the
+ // outer and inner TopN must still block expressions.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ newBlocked.addAll(buildOrderKeyExprIds(inner));
+ collectFromNode((Plan) inner.child(0), info, newBlocked);
+ return;
+ }
+
+ // Stop at boundary nodes that transform the schema or are data sources.
+ if (node instanceof LogicalRelation || node instanceof LogicalCTEProducer
+ || isBlockingNode(node)) {
+ return;
+ }
+
+ // Set operations are a boundary for the current TopN: do NOT collect
+ // expressions from below them. UNION ALL children may compute the same
+ // output column with different expressions (e.g. a+1 vs a+2), and a
+ // single pull-up Project above the TopN cannot represent branch-specific
+ // semantics. The normal visitor will still traverse into the children,
+ // so nested TopNs inside set operations are handled independently.
+ if (node instanceof LogicalSetOperation) {
+ return;
+ }
+
+ // For all other nodes, add their input slot ExprIds to the blocked set.
+ // Any operator that references a slot in its expressions prevents
+ // expressions that output that slot from being pulled up past it.
+ Set<ExprId> newBlocked = new HashSet<>(blockedExprIds);
+ for (Expression expr : node.getExpressions()) {
+ newBlocked.addAll(expr.getInputSlotExprIds());
+ if (expr instanceof NamedExpression) {
+ newBlocked.add(((NamedExpression) expr).getExprId());
+ }
+ }
+
+ for (Plan child : node.children()) {
+ collectFromNode(child, info, newBlocked);
+ }
+ }
+
+ // =========================================================================
+ // Pull-up eligibility
+ // =========================================================================
+
+ /**
+ * Check if a named expression can be pulled up above TopN.
+ * Eligible: Alias with non-trivial child, not blocked, no NoneMovableFunction.
+ */
+ static boolean canPullUp(NamedExpression ne) {
+ if (!(ne instanceof Alias)) {
+ return false;
+ }
+ Expression child = ((Alias) ne).child();
+ if (child instanceof Slot || child instanceof Literal) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof NoneMovableFunction)) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof Score)) {
+ return false;
+ }
+ if (ne.anyMatch(e -> e instanceof SlotReference && ((SlotReference) e).getDataType().isVariantType())) {
+ return false;
+ }
+ return true;
+ }
+
+ private static boolean isBlockingNode(Plan node) {
+ return node instanceof LogicalAggregate
+ || node instanceof LogicalWindow
+ || node instanceof LogicalRepeat;
+ }
+
+ private static Set<ExprId> buildOrderKeyExprIds(LogicalTopN<?> topN) {
+ Set<ExprId> orderKeyExprIds = new HashSet<>();
+ for (OrderKey orderKey : topN.getOrderKeys()) {
+ Expression keyExpr = orderKey.getExpr();
+ orderKeyExprIds.addAll(keyExpr.getInputSlotExprIds());
+ if (keyExpr instanceof NamedExpression) {
+ orderKeyExprIds.add(((NamedExpression) keyExpr).getExprId());
+ }
+ }
+ return orderKeyExprIds;
+ }
+
+ /**
+ * Deduplicate pull-up expressions so that each expression in a Project is only
+ * pulled up to the outermost TopN that collects it.
+ *
+ * <p>Since {@link CollectorContext#topNToPullUpInfo} is a {@link LinkedHashMap}
+ * and the Collector visits the plan top-down, iteration order is outer-to-inner.
+ * We keep the first occurrence of each (project-reference, exprId) pair and
+ * remove duplicates from inner TopNs.
+ */
+ private static void deduplicatePullUps(CollectorContext context) {
+ // Use IdentityHashMap because we need to distinguish Project nodes by object
+ // reference, not by content equality.
+ Map<LogicalProject<? extends Plan>, Set<ExprId>> handled = new IdentityHashMap<>();
+
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> toRemove = new ArrayList<>();
+ for (Map.Entry<LogicalProject<? extends Plan>, List<NamedExpression>> entry
+ : info.projectToPulledUpExprs.entrySet()) {
+ LogicalProject<? extends Plan> project = entry.getKey();
+ Set<ExprId> projectHandled = handled.computeIfAbsent(project, k -> new HashSet<>());
+ for (NamedExpression expr : entry.getValue()) {
+ if (projectHandled.contains(expr.getExprId())) {
+ toRemove.add(expr);
+ } else {
+ projectHandled.add(expr.getExprId());
+ }
+ }
+ }
+ for (NamedExpression expr : toRemove) {
+ info.addPassThroughExprForDeduplicatedExpr(expr);
+ info.allPulledUpExprs.remove(expr);
+ for (List<NamedExpression> list : info.projectToPulledUpExprs.values()) {
+ list.removeIf(e -> e == expr);
+ }
+ info.projectToPulledUpExprs.entrySet().removeIf(e -> e.getValue().isEmpty());
+ info.baseSlotsByExpr.remove(expr.getExprId());
+ }
+ }
+ }
+
+ // =========================================================================
+ // Pass 2: Replacer (bottom-up)
+ // =========================================================================
+
+ static class Replacer extends DefaultPlanRewriter<CollectorContext> {
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, CollectorContext context) {
+ LogicalProject<? extends Plan> rewritten = (LogicalProject<? extends Plan>) visit(project, context);
+
+ // Collect ALL pulled-up expressions across ALL PullUpInfos for this
+ // project. After dedup, each expression belongs to exactly one TopN
+ // (the outermost one that can pull it up). The project needs to be
+ // simplified by removing all of them, exposing their base slots once.
+ List<NamedExpression> allPulledUpExprs = collectAllPulledUpExprs(context, rewritten);
+ if (allPulledUpExprs.isEmpty() && rewritten != project
+ && rewritten.getProjects().equals(project.getProjects())) {
+ allPulledUpExprs = collectAllPulledUpExprs(context, project);
+ }
+ return simplifyProject(rewritten, allPulledUpExprs, context);
+ }
+
+ @Override
+ public Plan visitLogicalTopN(LogicalTopN topN, CollectorContext context) {
+ LogicalTopN rewritten = (LogicalTopN) visit(topN, context);
+ PullUpInfo info = context.getPullUpInfo(rewritten);
+ if (info == null && rewritten != topN) {
+ info = context.getPullUpInfo(topN);
+ }
+ if (info == null || (info.allPulledUpExprs.isEmpty()
+ && info.passThroughExprByDeduplicatedExpr.isEmpty())) {
+ return rewritten;
+ }
+ return addUpperProject(rewritten, info, context);
+ }
+ }
+
+ /**
+ * Collect all pulled-up expressions across all PullUpInfos for a project.
+ * After dedup each expression belongs to exactly one TopN, but the project
+ * must be simplified by removing all of them at once.
+ */
+ private static List<NamedExpression> collectAllPulledUpExprs(
+ CollectorContext context, LogicalProject<?> project) {
+ List<NamedExpression> result = new ArrayList<>();
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ List<NamedExpression> exprs = info.projectToPulledUpExprs.get(project);
+ if (exprs != null) {
+ result.addAll(exprs);
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Remove pulled-up expressions from this Project and add the input slots that still need to pass through TopN.
+ *
+ * <p>For example, after pulling up {@code x = a + 1}:
+ *
+ * <pre>
+ * TopN
+ * Project(id, x) -- forwards x from its child
+ * Project(id, a + 1 as x)
+ * Scan(id, a)
+ * </pre>
+ *
+ * <p>The lower Project should become {@code Project(id, a)}, because {@code x} is restored above TopN.
+ * The upper Project must also become {@code Project(id, a)} instead of keeping {@code Project(id, x)},
+ * since its child no longer outputs {@code x}.
+ */
+ private static LogicalProject<? extends Plan> simplifyProject(
+ LogicalProject<? extends Plan> project,
+ List<NamedExpression> pulledUpExprs,
+ CollectorContext context) {
+ Set<ExprId> childOutputExprIds = buildOutputExprIds((Plan) project.child(0));
+ List<Expression> passThroughExprs = collectUnavailablePullUpExprs(project, context, childOutputExprIds);
+ if (pulledUpExprs.isEmpty() && passThroughExprs.isEmpty()) {
+ return project;
+ }
+
+ Set<ExprId> pulledUpExprIds = new HashSet<>();
+ for (NamedExpression ne : pulledUpExprs) {
+ pulledUpExprIds.add(ne.getExprId());
+ }
+
+ List<NamedExpression> simplified = new ArrayList<>();
+ Set<ExprId> existingExprIds = new HashSet<>();
+ for (NamedExpression ne : project.getProjects()) {
+ if (!pulledUpExprIds.contains(ne.getExprId())
+ && !isUnavailablePullUpSlot(ne, context, childOutputExprIds)) {
+ NamedExpression resolved = resolveNamedExpression(ne, context, childOutputExprIds);
+ simplified.add(resolved);
+ existingExprIds.add(resolved.getExprId());
+ }
+ }
+
+ for (NamedExpression pulledUpExpr : pulledUpExprs) {
+ for (PullUpInfo info : context.topNToPullUpInfo.values()) {
+ if (info.baseSlotsByExpr.get(pulledUpExpr.getExprId()) != null) {
+ for (Slot baseSlot : resolveInputSlots(pulledUpExpr, context, childOutputExprIds)) {
+ if (!existingExprIds.contains(baseSlot.getExprId())) {
+ simplified.add(baseSlot);
+ existingExprIds.add(baseSlot.getExprId());
+ }
+ }
+ break; // found, no need to check other PullUpInfos
+ }
+ }
+ }
+ for (Expression passThroughExpr : passThroughExprs) {
+ for (Slot baseSlot : resolveInputSlots(passThroughExpr, context, childOutputExprIds)) {
+ if (!existingExprIds.contains(baseSlot.getExprId())) {
+ simplified.add(baseSlot);
+ existingExprIds.add(baseSlot.getExprId());
+ }
+ }
+ }
+
+ if (simplified.equals(project.getProjects())) {
+ return project;
+ }
+ return (LogicalProject<? extends Plan>) project.withProjects(simplified);
+ }
+
+ private static List<Expression> collectUnavailablePullUpExprs(
+ LogicalProject<? extends Plan> project, CollectorContext context, Set<ExprId> childOutputExprIds) {
+ List<Expression> passThroughExprs = new ArrayList<>();
+ for (NamedExpression ne : project.getProjects()) {
+ if (isUnavailablePullUpSlot(ne, context, childOutputExprIds)) {
+ passThroughExprs.add(getPullUpReplaceExpression((Slot) ne, context));
+ }
+ }
+ return passThroughExprs;
+ }
+
+ private static boolean isUnavailablePullUpSlot(
+ NamedExpression ne, CollectorContext context, Set<ExprId> childOutputExprIds) {
+ return ne instanceof Slot
+ && !childOutputExprIds.contains(ne.getExprId())
+ && getPullUpReplaceExpression((Slot) ne, context) != null;
+ }
+
+ private static Expression getPullUpReplaceExpression(Slot slot, CollectorContext context) {
+ for (Map.Entry<Slot, Expression> entry : context.pullUpExprReplaceMap.entrySet()) {
+ if (entry.getKey().getExprId().equals(slot.getExprId())) {
+ return entry.getValue();
+ }
+ }
+ return null;
+ }
+
+ /** Create a new Project above the TopN that restores pulled-up expressions. */
+ private static LogicalProject<Plan> addUpperProject(LogicalTopN topN, PullUpInfo info,
+ CollectorContext context) {
+ Map<ExprId, NamedExpression> pulledUpBySlotExprId = new HashMap<>();
+ Set<ExprId> currentOutputExprIds = buildOutputExprIds(topN);
+ for (NamedExpression e : info.allPulledUpExprs) {
+ pulledUpBySlotExprId.put(e.toSlot().getExprId(), resolvePulledUpExpr(e, context, currentOutputExprIds));
+ }
+
+ // Use the current (possibly rewritten) TopN's output so that slots
+ // whose expressions were deduplicated to an outer TopN reference
+ // the correct post-simplification ExprIds instead of stale ones.
+ List<Slot> currentOutput = topN.getOutput();
+ Map<ExprId, Slot> currentOutputByExprId = new HashMap<>();
+ for (Slot slot : currentOutput) {
+ currentOutputByExprId.put(slot.getExprId(), slot);
+ }
+ List<NamedExpression> upperOutput = new ArrayList<>();
+ Set<ExprId> upperOutputExprIds = new HashSet<>();
+ Set<ExprId> passThroughOutputExprIds = new HashSet<>();
+ for (int i = 0; i < info.originalTopNOutput.size(); i++) {
+ Slot origSlot = info.originalTopNOutput.get(i);
+ NamedExpression pulledUpExpr = pulledUpBySlotExprId.get(origSlot.getExprId());
+ if (pulledUpExpr != null) {
+ upperOutput.add(pulledUpExpr);
+ upperOutputExprIds.add(pulledUpExpr.getExprId());
+ addPassThroughSlots(upperOutput, upperOutputExprIds, passThroughOutputExprIds,
Review Comment:
Adding the pulled expression's input slots to `upperOutput` makes these pass-through columns part of the `LogicalProject` output. For a non-nested case such as `Project(x=a+1, b, id) -> TopN(order by id)`, the lower project is simplified to pass `a`, and this branch rebuilds the upper project as `[x, a, b, id]` instead of the original `[x, b, id]`; the new/updated unit tests even expect this extra column. Because this project can be the query root (or feed the result sink), the rewrite changes the user-visible schema and row arity. Please keep pass-through slots available below the TopN, but do not include them in the restored user-visible upper project output unless there is an outer operator that actually requires them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4633652685
<details>
<summary>TPC-H: <b>Total hot run time: 29441 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9019aef33ce3081fa110c40f4cc67cb0236c0556, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17670 4034 4027 4027
q2 q3 10805 1459 832 832
q4 4686 502 358 358
q5 7563 874 586 586
q6 184 175 140 140
q7 802 817 639 639
q8 9408 1633 1583 1583
q9 5784 4486 4514 4486
q10 6777 1815 1549 1549
q11 444 276 260 260
q12 635 432 295 295
q13 18156 3381 2769 2769
q14 271 263 243 243
q15 q16 827 776 700 700
q17 1003 959 911 911
q18 6912 5871 5589 5589
q19 1753 1373 1058 1058
q20 530 427 267 267
q21 6464 2825 2827 2825
q22 460 377 324 324
Total cold run time: 101134 ms
Total hot run time: 29441 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5138 4897 5018 4897
q2 q3 4875 5282 4664 4664
q4 2142 2169 1384 1384
q5 4976 4922 4816 4816
q6 238 183 136 136
q7 1876 1740 1651 1651
q8 2580 2224 2217 2217
q9 8107 8092 7477 7477
q10 4745 4846 4229 4229
q11 537 381 350 350
q12 722 739 523 523
q13 3067 3415 2761 2761
q14 272 272 247 247
q15 q16 680 698 615 615
q17 1300 1286 1262 1262
q18 7564 6841 6926 6841
q19 1161 1108 1115 1108
q20 2210 2220 1934 1934
q21 5297 4728 4442 4442
q22 548 449 413 413
Total cold run time: 58035 ms
Total hot run time: 51967 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4624198376
<details>
<summary>TPC-DS: <b>Total hot run time: 169256 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c005743605d8bb37f81400c7996fe80d09a8875, data reload: false
query5 4326 613 492 492
query6 444 199 182 182
query7 4810 541 319 319
query8 363 220 211 211
query9 8791 4017 4044 4017
query10 470 317 269 269
query11 5863 2366 2186 2186
query12 162 102 109 102
query13 1283 636 425 425
query14 6419 5445 5084 5084
query14_1 4416 4427 4406 4406
query15 212 196 181 181
query16 1056 466 448 448
query17 1131 730 604 604
query18 2622 484 360 360
query19 211 188 149 149
query20 113 114 110 110
query21 220 145 125 125
query22 13693 13633 13476 13476
query23 17327 16528 16178 16178
query23_1 16291 16298 16308 16298
query24 7482 1771 1305 1305
query24_1 1349 1322 1329 1322
query25 572 482 423 423
query26 1292 325 173 173
query27 2641 567 346 346
query28 4431 2053 2015 2015
query29 1103 682 513 513
query30 315 241 206 206
query31 1148 1082 956 956
query32 116 63 61 61
query33 552 327 262 262
query34 1183 1100 696 696
query35 746 799 679 679
query36 1358 1400 1257 1257
query37 153 104 91 91
query38 3228 3148 3035 3035
query39 920 921 918 918
query39_1 876 870 878 870
query40 223 120 98 98
query41 64 61 62 61
query42 95 97 94 94
query43 320 321 278 278
query44
query45 197 187 183 183
query46 1137 1211 765 765
query47 2393 2409 2255 2255
query48 393 413 281 281
query49 633 473 362 362
query50 1015 343 259 259
query51 4359 4266 4239 4239
query52 89 91 78 78
query53 253 279 196 196
query54 271 222 219 219
query55 77 75 70 70
query56 238 254 222 222
query57 1429 1407 1300 1300
query58 257 218 203 203
query59 1633 1687 1508 1508
query60 275 248 234 234
query61 158 156 155 155
query62 737 653 581 581
query63 234 188 191 188
query64 2531 811 625 625
query65
query66 1754 455 341 341
query67 29767 29741 29536 29536
query68
query69 412 311 273 273
query70 970 952 958 952
query71 306 223 204 204
query72 2960 2650 2382 2382
query73 842 793 432 432
query74 5181 4955 4800 4800
query75 2651 2586 2244 2244
query76 2371 1140 739 739
query77 348 402 291 291
query78 12287 12560 11809 11809
query79 1437 1083 793 793
query80 827 476 396 396
query81 502 281 247 247
query82 586 160 123 123
query83 344 271 258 258
query84
query85 911 522 434 434
query86 409 314 275 275
query87 3381 3391 3189 3189
query88 3665 2777 2738 2738
query89 425 390 329 329
query90 1778 175 175 175
query91 174 163 137 137
query92 62 58 57 57
query93 1656 1407 921 921
query94 620 355 319 319
query95 664 489 339 339
query96 1051 769 358 358
query97 2692 2694 2551 2551
query98 213 205 200 200
query99 1171 1163 1034 1034
Total cold run time: 251566 ms
Total hot run time: 169256 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4563264052
<details>
<summary>TPC-H: <b>Total hot run time: 31456 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80a6ff0127dca6e6fe6f5ed9f4e0477ab0281d2c, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17611 4093 4047 4047
q2 q3 10764 1404 848 848
q4 4687 474 345 345
q5 7600 2302 2126 2126
q6 251 180 142 142
q7 1011 761 676 676
q8 9367 1634 1620 1620
q9 6470 4992 4957 4957
q10 6447 2204 1893 1893
q11 432 283 247 247
q12 695 435 304 304
q13 18216 3396 2813 2813
q14 271 257 238 238
q15 q16 820 780 720 720
q17 998 957 1001 957
q18 6959 5730 5597 5597
q19 1291 1339 977 977
q20 516 418 269 269
q21 5669 2607 2386 2386
q22 439 353 294 294
Total cold run time: 100514 ms
Total hot run time: 31456 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4407 4413 4438 4413
q2 q3 4524 4943 4363 4363
q4 2105 2246 1393 1393
q5 4465 4306 4760 4306
q6 263 201 151 151
q7 1935 1929 1676 1676
q8 2635 2256 2268 2256
q9 7993 7875 8094 7875
q10 4804 4820 4346 4346
q11 578 423 397 397
q12 804 739 548 548
q13 3284 3751 2984 2984
q14 310 313 275 275
q15 q16 735 733 644 644
q17 1396 1357 1357 1357
q18 8031 7329 7136 7136
q19 1129 1107 1100 1100
q20 2228 2225 1941 1941
q21 5302 4615 4503 4503
q22 539 484 423 423
Total cold run time: 57467 ms
Total hot run time: 52087 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4563202907
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #63736:
URL: https://github.com/apache/doris/pull/63736#discussion_r3319061515
##########
regression-test/suites/nereids_rules_p0/column_pruning/topn_expr_pullup.groovy:
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("topn_expr_pullup") {
+ sql """ set topn_lazy_materialization_threshold=1024; """
+ sql """ DROP TABLE IF EXISTS tep_tbl """
+ sql """
+ CREATE TABLE tep_tbl (
+ id INT,
+ str_col STRING NULL,
+ struct_col STRUCT<city: STRING, zip: INT> NULL,
+ arr_col ARRAY<INT> NULL,
+ map_col MAP<STRING, INT> NULL,
+ int_col INT NULL
+ ) ENGINE = OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1")
+ """
+ sql """ INSERT INTO tep_tbl VALUES
+ (1, 'hello', named_struct('city', 'NYC', 'zip', 10001), [1, 2, 3], {'a': 1, 'b': 2}, 10),
+ (2, 'world', named_struct('city', 'LA', 'zip', 90001), [4, 5], {'x': 3}, 20),
+ (3, 'doris', named_struct('city', 'SF', 'zip', 94101), [6], {'y': 5}, 30)
+ """
+
+ // =============================================
+ // Test 1: STRUCT PPD — expression pulled above TopN + lazy mat
+ // =============================================
+ explain {
+ sql """ select id, struct_element(struct_col, 'city') as city
+ from tep_tbl order by id limit 3 """
+ contains("VMaterializeNode")
+ contains("row_ids: [__DORIS_GLOBAL_ROWID_COL__tep_tbl]")
+ }
+ qt_struct_ppd """ select id, struct_element(struct_col, 'city') as city
Review Comment:
This new suite introduces multiple `qt_` blocks, but there is no corresponding `regression-test/data/nereids_rules_p0/column_pruning/topn_expr_pullup.out` file in the PR. The regression runner expects every `qt_` label to have generated output, so this suite will fail before validating the new TopN expression pull-up behavior. Please run the regression case and add the generated `.out` file for all labels in this suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4565923291
<details>
<summary>TPC-H: <b>Total hot run time: 32411 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2315196727fc105b516bd6d181ac447bc29f8e8, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17679 4194 4169 4169
q2 q3 10794 1426 850 850
q4 4687 491 336 336
q5 7538 2441 2193 2193
q6 250 189 140 140
q7 976 791 632 632
q8 9359 1696 1557 1557
q9 5287 5048 5056 5048
q10 6359 2191 1864 1864
q11 439 291 261 261
q12 635 458 299 299
q13 18193 3496 2807 2807
q14 266 258 228 228
q15 q16 816 775 710 710
q17 1016 1043 1039 1039
q18 7019 5593 5577 5577
q19 1298 1368 1278 1278
q20 561 452 271 271
q21 6414 2904 2843 2843
q22 451 371 309 309
Total cold run time: 100037 ms
Total hot run time: 32411 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5145 5006 4990 4990
q2 q3 4986 5370 4592 4592
q4 2121 2243 1382 1382
q5 5119 4780 4788 4780
q6 260 197 133 133
q7 1941 1728 1534 1534
q8 2574 2260 2301 2260
q9 7713 7566 7501 7501
q10 4790 4684 4244 4244
q11 545 376 346 346
q12 738 754 531 531
q13 3082 3477 2837 2837
q14 279 291 267 267
q15 q16 678 698 592 592
q17 1294 1276 1271 1271
q18 7529 7003 6916 6916
q19 1092 1088 1152 1088
q20 2236 2213 1926 1926
q21 5324 4615 4497 4497
q22 507 455 401 401
Total cold run time: 57953 ms
Total hot run time: 52088 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4566013891
<details>
<summary>TPC-DS: <b>Total hot run time: 171814 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2315196727fc105b516bd6d181ac447bc29f8e8, data reload: false
query5 4314 663 522 522
query6 337 230 213 213
query7 4225 600 310 310
query8 333 244 229 229
query9 8795 4102 4077 4077
query10 457 361 306 306
query11 5713 2482 2293 2293
query12 186 130 123 123
query13 1274 609 439 439
query14 6120 5463 5194 5194
query14_1 4490 4480 4471 4471
query15 214 207 192 192
query16 1051 470 462 462
query17 1194 720 582 582
query18 2716 482 355 355
query19 223 199 168 168
query20 136 130 128 128
query21 217 144 119 119
query22 13694 13562 13453 13453
query23 17334 16432 16218 16218
query23_1 16394 16320 16383 16320
query24 7420 1793 1323 1323
query24_1 1318 1313 1326 1313
query25 584 473 404 404
query26 1300 326 175 175
query27 2683 559 354 354
query28 4408 2030 2010 2010
query29 970 622 495 495
query30 308 243 199 199
query31 1136 1081 953 953
query32 93 74 74 74
query33 529 352 303 303
query34 1175 1170 663 663
query35 780 786 703 703
query36 1369 1458 1257 1257
query37 150 107 93 93
query38 3221 3167 3120 3120
query39 929 938 907 907
query39_1 880 885 868 868
query40 236 145 123 123
query41 67 63 63 63
query42 142 107 113 107
query43 323 338 284 284
query44
query45 211 205 195 195
query46 1105 1183 766 766
query47 2372 2327 2196 2196
query48 407 406 294 294
query49 643 496 382 382
query50 987 350 261 261
query51 4338 4289 4285 4285
query52 102 107 100 100
query53 253 282 214 214
query54 313 279 252 252
query55 93 93 90 90
query56 300 293 314 293
query57 1430 1408 1324 1324
query58 297 276 258 258
query59 1592 1690 1451 1451
query60 319 323 307 307
query61 160 158 153 153
query62 700 648 586 586
query63 245 197 212 197
query64 2395 814 638 638
query65
query66 1690 484 370 370
query67 29679 29656 29550 29550
query68
query69 475 349 302 302
query70 1002 1014 1041 1014
query71 317 278 268 268
query72 3003 2726 2436 2436
query73 847 747 437 437
query74 5141 5013 4842 4842
query75 2701 2591 2269 2269
query76 2299 1178 785 785
query77 414 420 343 343
query78 12365 12326 11815 11815
query79 1277 1073 793 793
query80 613 543 458 458
query81 444 283 250 250
query82 240 159 127 127
query83 277 280 247 247
query84
query85 847 527 444 444
query86 357 346 309 309
query87 3463 3399 3284 3284
query88 3676 2756 2742 2742
query89 419 395 342 342
query90 2190 189 177 177
query91 177 163 139 139
query92 84 77 78 77
query93 1465 1469 959 959
query94 552 361 312 312
query95 678 465 363 363
query96 1052 786 351 351
query97 2734 2734 2620 2620
query98 236 228 227 227
query99 1158 1164 1030 1030
Total cold run time: 252731 ms
Total hot run time: 171814 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4566114825
<details>
<summary>TPC-DS: <b>Total hot run time: 170755 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 835b604c4ad63b88f63c6caeb098d12c98a4be08, data reload: false
query5 4327 680 532 532
query6 328 229 196 196
query7 4258 615 322 322
query8 334 227 223 223
query9 8838 4037 4038 4037
query10 470 344 298 298
query11 5768 2546 2184 2184
query12 180 128 126 126
query13 1302 598 451 451
query14 6111 5451 5162 5162
query14_1 4461 4492 4462 4462
query15 216 207 183 183
query16 992 457 467 457
query17 1187 726 608 608
query18 2477 488 358 358
query19 225 222 165 165
query20 138 133 128 128
query21 221 141 116 116
query22 13715 13563 13312 13312
query23 17223 16511 16116 16116
query23_1 16314 16349 16191 16191
query24 7487 1776 1335 1335
query24_1 1314 1337 1324 1324
query25 559 484 400 400
query26 1316 304 168 168
query27 2751 563 346 346
query28 4450 1970 1961 1961
query29 1001 635 477 477
query30 307 242 203 203
query31 1120 1087 947 947
query32 93 77 71 71
query33 539 344 285 285
query34 1192 1119 644 644
query35 767 783 709 709
query36 1387 1405 1236 1236
query37 149 105 91 91
query38 3217 3157 3060 3060
query39 944 913 895 895
query39_1 876 864 868 864
query40 230 151 125 125
query41 64 62 61 61
query42 111 108 118 108
query43 332 333 293 293
query44
query45 212 206 190 190
query46 1073 1163 743 743
query47 2383 2401 2215 2215
query48 402 396 297 297
query49 622 488 372 372
query50 959 357 242 242
query51 4357 4393 4280 4280
query52 109 101 94 94
query53 253 279 200 200
query54 308 265 253 253
query55 90 90 89 89
query56 296 291 297 291
query57 1433 1414 1316 1316
query58 297 275 251 251
query59 1545 1647 1455 1455
query60 326 319 303 303
query61 166 184 181 181
query62 706 662 589 589
query63 246 206 213 206
query64 2460 849 700 700
query65
query66 1741 497 382 382
query67 29708 29740 29459 29459
query68
query69 467 352 321 321
query70 1039 979 988 979
query71 307 280 275 275
query72 3143 2685 2373 2373
query73 864 739 420 420
query74 5134 4924 4722 4722
query75 2679 2614 2280 2280
query76 2274 1116 765 765
query77 403 402 327 327
query78 12582 12506 11908 11908
query79 1448 1031 773 773
query80 637 532 451 451
query81 454 277 238 238
query82 1366 157 127 127
query83 354 291 251 251
query84
query85 891 585 458 458
query86 398 321 323 321
query87 3506 3400 3234 3234
query88 3613 2711 2682 2682
query89 436 408 351 351
query90 1990 182 174 174
query91 177 163 137 137
query92 79 77 70 70
query93 1571 1427 853 853
query94 537 346 312 312
query95 665 386 339 339
query96 1067 755 366 366
query97 2751 2765 2596 2596
query98 232 229 247 229
query99 1173 1162 1022 1022
Total cold run time: 254187 ms
Total hot run time: 170755 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4617875785
<details>
<summary>TPC-DS: <b>Total hot run time: 169688 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa8526456fd8a1e1a2f7800f79ffb445388e4953, data reload: false
query5 4319 660 492 492
query6 427 199 177 177
query7 4818 543 299 299
query8 388 217 209 209
query9 8761 4038 4014 4014
query10 451 320 263 263
query11 5877 2367 2228 2228
query12 157 102 96 96
query13 1259 617 438 438
query14 6420 5423 5125 5125
query14_1 4388 4455 4407 4407
query15 208 196 176 176
query16 1027 453 413 413
query17 1103 691 578 578
query18 2439 476 342 342
query19 203 177 140 140
query20 116 112 105 105
query21 213 139 120 120
query22 13706 13596 13431 13431
query23 17487 16532 16217 16217
query23_1 16334 16275 16337 16275
query24 7620 1779 1297 1297
query24_1 1331 1320 1289 1289
query25 543 452 383 383
query26 1333 305 159 159
query27 2771 605 337 337
query28 4522 2028 2009 2009
query29 1121 638 540 540
query30 310 236 200 200
query31 1118 1073 963 963
query32 101 60 60 60
query33 512 324 255 255
query34 1181 1134 664 664
query35 755 790 715 715
query36 1389 1407 1221 1221
query37 161 111 94 94
query38 3206 3174 3068 3068
query39 945 942 909 909
query39_1 886 889 880 880
query40 222 122 102 102
query41 63 61 61 61
query42 96 95 93 93
query43 318 323 276 276
query44
query45 200 191 181 181
query46 1094 1183 784 784
query47 2363 2395 2276 2276
query48 401 428 276 276
query49 621 479 361 361
query50 1027 371 255 255
query51 4343 4316 4205 4205
query52 89 90 79 79
query53 251 270 188 188
query54 272 212 195 195
query55 81 75 69 69
query56 241 244 228 228
query57 1418 1397 1294 1294
query58 241 212 208 208
query59 1572 1636 1409 1409
query60 277 255 229 229
query61 162 156 152 152
query62 714 658 629 629
query63 232 195 195 195
query64 2579 800 634 634
query65
query66 1799 464 352 352
query67 30100 29639 29442 29442
query68
query69 435 305 275 275
query70 1013 969 965 965
query71 306 219 207 207
query72 2953 2661 2491 2491
query73 857 789 443 443
query74 5141 4982 4839 4839
query75 2703 2615 2271 2271
query76 2322 1175 805 805
query77 371 388 307 307
query78 12304 12365 11891 11891
query79 1233 1025 786 786
query80 579 513 436 436
query81 452 288 246 246
query82 232 163 125 125
query83 283 285 259 259
query84
query85 948 629 525 525
query86 342 300 272 272
query87 3397 3389 3191 3191
query88 3699 2783 2792 2783
query89 407 383 339 339
query90 2207 188 175 175
query91 175 166 140 140
query92 67 63 58 58
query93 1414 1391 876 876
query94 539 356 316 316
query95 699 384 347 347
query96 1003 793 360 360
query97 2674 2665 2568 2568
query98 210 207 208 207
query99 1145 1184 1026 1026
Total cold run time: 251192 ms
Total hot run time: 169688 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](fe) TopN lazy materialization support struct/variant nested column pruning [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63736:
URL: https://github.com/apache/doris/pull/63736#issuecomment-4604103918
# FE UT Coverage Report
Increment line coverage ` 72.03% (327/454)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63736_14da24479173944ea5601dd5c550fa43e7079e0d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63736_14da24479173944ea5601dd5c550fa43e7079e0d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org