You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "englefly (via GitHub)" <gi...@apache.org> on 2026/05/16 23:54:59 UTC
[PR] [opt](nereids) optimize InferAggNotNull rule [doris]
englefly opened a new pull request, #63318:
URL: https://github.com/apache/doris/pull/63318
### What problem does this PR solve?
Optimize InferAggNotNull aggregate inference
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Problem Summary: Avoid repeated aggregate-function collection in InferAggNotNull, stop traversing aggregate arguments while collecting aggregate functions, add complexity guards for not-null inference, and safely support multi-aggregate cases by only pushing predicates common to every aggregate function.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4539768498
# FE Regression Coverage Report
Increment line coverage ` 84.81% (67/79)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_b2147a5ba9b189ebbbe472cb89b46f3e62357f12_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_b2147a5ba9b189ebbbe472cb89b46f3e62357f12_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4526979667
<details>
<summary>TPC-H: <b>Total hot run time: 31711 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa63bde8dfa91234456fdd7a5b284c9617eec400, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17637 4172 4077 4077
q2 q3 10812 1361 798 798
q4 4679 474 343 343
q5 7782 2299 2128 2128
q6 261 177 136 136
q7 957 792 635 635
q8 9394 1679 1588 1588
q9 6462 4991 4986 4986
q10 6432 2217 1904 1904
q11 434 273 255 255
q12 690 428 301 301
q13 18184 3461 2773 2773
q14 274 260 246 246
q15 q16 828 771 708 708
q17 989 979 945 945
q18 6826 5765 5539 5539
q19 1182 1239 1197 1197
q20 558 423 283 283
q21 6136 2762 2536 2536
q22 472 383 333 333
Total cold run time: 100989 ms
Total hot run time: 31711 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5145 4765 5085 4765
q2 q3 4890 5276 4684 4684
q4 2193 2238 1428 1428
q5 5061 4861 4812 4812
q6 233 188 140 140
q7 1963 1807 1572 1572
q8 2276 2039 2003 2003
q9 7522 7465 7526 7465
q10 4772 4672 4237 4237
q11 576 399 401 399
q12 741 748 551 551
q13 3031 3355 2754 2754
q14 272 284 269 269
q15 q16 675 705 622 622
q17 1304 1291 1286 1286
q18 7430 6920 6838 6838
q19 1141 1088 1107 1088
q20 2239 2239 1974 1974
q21 5387 4648 4624 4624
q22 539 474 418 418
Total cold run time: 57390 ms
Total hot run time: 51929 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) optimize InferAggNotNull rule [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4476433404
# FE Regression Coverage Report
Increment line coverage ` 31.88% (44/138)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_5e656119a9407c4c26e6d0479a04583ae050412b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_5e656119a9407c4c26e6d0479a04583ae050412b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4483253651
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4526922024
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4532657626
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) optimize InferAggNotNull rule [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4475157328
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4538173048
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) optimize InferAggNotNull rule [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4468569170
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4478623782
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #63318:
URL: https://github.com/apache/doris/pull/63318#discussion_r3279100361
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNull.java:
##########
@@ -47,19 +49,10 @@ public class InferAggNotNull extends OneRewriteRuleFactory {
public Rule build() {
return logicalAggregate()
.when(agg -> agg.getGroupByExpressions().size() == 0)
- .when(agg -> agg.getAggregateFunctions().size() == 1)
- .when(agg -> {
- Set<AggregateFunction> funcs = agg.getAggregateFunctions();
- return funcs.stream().allMatch(f -> f instanceof Count)
- || funcs.stream().allMatch(f -> f instanceof Avg)
- || funcs.stream().allMatch(f -> f instanceof Sum)
- || funcs.stream().allMatch(f -> f instanceof Max)
- || funcs.stream().allMatch(f -> f instanceof Min);
- }).thenApply(ctx -> {
+ .thenApply(ctx -> {
LogicalAggregate<Plan> agg = ctx.root;
- Set<Expression> exprs = agg.getAggregateFunctions().stream().flatMap(f -> f.children().stream())
Review Comment:
good idea. It is done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4505392185
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4478029948
<details>
<summary>TPC-H: <b>Total hot run time: 31522 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ba0872059324be0298b91f6e90e9fddb3541f06, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17599 3983 3992 3983
q2 q3 10771 1407 808 808
q4 4686 481 349 349
q5 7542 2293 2117 2117
q6 247 193 147 147
q7 983 787 635 635
q8 9429 1713 1607 1607
q9 5157 4898 4906 4898
q10 6380 2060 1790 1790
q11 428 273 248 248
q12 632 424 309 309
q13 18133 3374 2782 2782
q14 263 259 241 241
q15 q16 829 771 711 711
q17 1009 917 931 917
q18 6985 5681 5658 5658
q19 1311 1270 986 986
q20 502 465 360 360
q21 6294 2928 2653 2653
q22 475 390 323 323
Total cold run time: 99655 ms
Total hot run time: 31522 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 5101 4809 4701 4701
q2 q3 4925 5285 4658 4658
q4 2202 2235 1401 1401
q5 4765 4877 4691 4691
q6 232 179 137 137
q7 1845 1782 1560 1560
q8 2460 2093 2189 2093
q9 7760 7673 7275 7275
q10 4493 4380 3994 3994
q11 522 388 395 388
q12 715 723 522 522
q13 3023 3412 2792 2792
q14 275 277 254 254
q15 q16 689 701 627 627
q17 1309 1247 1246 1246
q18 7298 6700 6949 6700
q19 1092 1085 1094 1085
q20 2250 2201 1951 1951
q21 5345 4687 4483 4483
q22 514 497 427 427
Total cold run time: 56815 ms
Total hot run time: 50985 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4478624707
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4483715786
# FE Regression Coverage Report
Increment line coverage ` 27.08% (75/277)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_8ba0872059324be0298b91f6e90e9fddb3541f06_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_8ba0872059324be0298b91f6e90e9fddb3541f06_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4538700682
# FE UT Coverage Report
Increment line coverage ` 85.53% (65/76)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_aa63bde8dfa91234456fdd7a5b284c9617eec400/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_aa63bde8dfa91234456fdd7a5b284c9617eec400/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4538829252
<details>
<summary>TPC-DS: <b>Total hot run time: 173005 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2147a5ba9b189ebbbe472cb89b46f3e62357f12, data reload: false
query5 4301 666 515 515
query6 337 222 199 199
query7 4225 597 313 313
query8 319 237 228 228
query9 8848 4236 4231 4231
query10 461 349 317 317
query11 5786 2638 2276 2276
query12 184 130 125 125
query13 1291 605 448 448
query14 6137 5570 5232 5232
query14_1 4583 4573 4519 4519
query15 214 203 188 188
query16 994 446 450 446
query17 1172 755 612 612
query18 2612 505 369 369
query19 218 205 168 168
query20 147 137 135 135
query21 236 148 121 121
query22 13658 13564 13387 13387
query23 17499 16694 16451 16451
query23_1 16605 16409 16546 16409
query24 7393 1782 1340 1340
query24_1 1330 1340 1328 1328
query25 571 507 442 442
query26 1320 322 181 181
query27 2737 540 350 350
query28 4430 2062 2042 2042
query29 1037 641 521 521
query30 321 251 201 201
query31 1129 1092 945 945
query32 91 79 77 77
query33 544 370 309 309
query34 1185 1194 648 648
query35 780 831 715 715
query36 1461 1397 1239 1239
query37 149 103 90 90
query38 3217 3163 3084 3084
query39 928 919 891 891
query39_1 875 880 874 874
query40 230 144 125 125
query41 66 63 61 61
query42 109 106 111 106
query43 333 336 293 293
query44
query45 214 205 195 195
query46 1070 1160 742 742
query47 2372 2409 2330 2330
query48 384 398 314 314
query49 623 496 388 388
query50 1021 359 265 265
query51 4407 4326 4331 4326
query52 105 107 93 93
query53 255 281 202 202
query54 313 283 251 251
query55 96 87 84 84
query56 296 315 300 300
query57 1443 1438 1355 1355
query58 307 276 273 273
query59 1611 1622 1418 1418
query60 316 330 308 308
query61 154 157 159 157
query62 694 656 589 589
query63 238 204 207 204
query64 2434 799 640 640
query65
query66 1705 489 354 354
query67 29683 29634 29588 29588
query68
query69 466 345 325 325
query70 1059 1043 1013 1013
query71 308 274 271 271
query72 3112 2711 2391 2391
query73 836 767 418 418
query74 5119 4938 4806 4806
query75 2706 2611 2280 2280
query76 2289 1169 768 768
query77 401 415 341 341
query78 12509 12466 11925 11925
query79 1486 1012 760 760
query80 1337 547 450 450
query81 510 279 239 239
query82 1251 160 127 127
query83 361 284 247 247
query84 260 146 108 108
query85 931 546 453 453
query86 477 348 344 344
query87 3476 3421 3270 3270
query88 3624 2781 2729 2729
query89 448 392 344 344
query90 1917 172 185 172
query91 180 172 144 144
query92 83 76 75 75
query93 1529 1371 831 831
query94 741 311 313 311
query95 676 480 353 353
query96 1053 848 345 345
query97 2756 2788 2655 2655
query98 244 231 232 231
query99 1179 1224 1048 1048
Total cold run time: 256181 ms
Total hot run time: 173005 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly merged PR #63318:
URL: https://github.com/apache/doris/pull/63318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4532657976
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on code in PR #63318:
URL: https://github.com/apache/doris/pull/63318#discussion_r3279099525
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNull.java:
##########
@@ -47,19 +49,10 @@ public class InferAggNotNull extends OneRewriteRuleFactory {
public Rule build() {
return logicalAggregate()
.when(agg -> agg.getGroupByExpressions().size() == 0)
- .when(agg -> agg.getAggregateFunctions().size() == 1)
- .when(agg -> {
- Set<AggregateFunction> funcs = agg.getAggregateFunctions();
- return funcs.stream().allMatch(f -> f instanceof Count)
- || funcs.stream().allMatch(f -> f instanceof Avg)
- || funcs.stream().allMatch(f -> f instanceof Sum)
- || funcs.stream().allMatch(f -> f instanceof Max)
- || funcs.stream().allMatch(f -> f instanceof Min);
- }).thenApply(ctx -> {
+ .thenApply(ctx -> {
LogicalAggregate<Plan> agg = ctx.root;
- Set<Expression> exprs = agg.getAggregateFunctions().stream().flatMap(f -> f.children().stream())
- .collect(Collectors.toSet());
- Set<Expression> isNotNulls = ExpressionUtils.inferNotNull(exprs, ctx.cascadesContext);
+ Set<AggregateFunction> aggregateFunctions = collectAggregateFunctions(agg);
+ Set<Expression> isNotNulls = inferCommonNotNulls(aggregateFunctions, ctx.cascadesContext);
Review Comment:
ExpressionUtils.inferNotNull is invoked in side inferCommonNotNulls
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) optimize InferAggNotNull rule [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4476206502
<details>
<summary>TPC-DS: <b>Total hot run time: 169737 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5e656119a9407c4c26e6d0479a04583ae050412b, data reload: false
query5 4366 648 499 499
query6 355 233 198 198
query7 4236 543 302 302
query8 326 224 216 216
query9 8817 4090 4035 4035
query10 446 337 289 289
query11 5804 2420 2180 2180
query12 178 127 120 120
query13 1334 597 422 422
query14 6007 5375 5057 5057
query14_1 4363 4348 4355 4348
query15 224 209 187 187
query16 974 466 458 458
query17 1137 765 610 610
query18 2578 493 366 366
query19 227 205 167 167
query20 144 139 129 129
query21 224 143 120 120
query22 13595 13476 13381 13381
query23 17174 16335 15995 15995
query23_1 16187 16121 16129 16121
query24 7587 1782 1261 1261
query24_1 1303 1313 1315 1313
query25 565 502 429 429
query26 1334 322 174 174
query27 2694 564 353 353
query28 4425 1941 1931 1931
query29 1001 636 521 521
query30 315 237 198 198
query31 1107 1067 936 936
query32 92 77 78 77
query33 558 365 306 306
query34 1172 1102 642 642
query35 764 802 699 699
query36 1331 1357 1169 1169
query37 156 100 87 87
query38 3203 3121 3064 3064
query39 926 917 913 913
query39_1 866 867 885 867
query40 236 147 122 122
query41 64 63 64 63
query42 109 108 108 108
query43 327 331 283 283
query44
query45 213 198 196 196
query46 1055 1169 735 735
query47 2339 2371 2251 2251
query48 382 431 295 295
query49 629 491 381 381
query50 1001 346 259 259
query51 4283 4312 4258 4258
query52 106 103 97 97
query53 250 283 207 207
query54 311 263 247 247
query55 92 90 85 85
query56 304 316 313 313
query57 1437 1418 1332 1332
query58 300 271 269 269
query59 1544 1619 1407 1407
query60 321 326 305 305
query61 195 155 157 155
query62 670 628 554 554
query63 238 201 215 201
query64 2408 816 646 646
query65
query66 1709 465 358 358
query67 30016 29918 29909 29909
query68
query69 471 337 307 307
query70 1057 1016 1027 1016
query71 298 274 270 270
query72 2997 2703 2373 2373
query73 874 744 436 436
query74 5080 4882 4731 4731
query75 2666 2608 2234 2234
query76 2286 1136 722 722
query77 398 442 338 338
query78 12115 12234 11635 11635
query79 1455 1045 684 684
query80 649 535 450 450
query81 455 279 237 237
query82 1385 160 123 123
query83 359 272 249 249
query84 265 134 113 113
query85 909 557 454 454
query86 384 344 337 337
query87 3415 3381 3177 3177
query88 3515 2638 2611 2611
query89 438 382 339 339
query90 1977 185 179 179
query91 175 179 139 139
query92 84 88 76 76
query93 1502 1586 842 842
query94 545 335 327 327
query95 666 373 432 373
query96 1027 819 323 323
query97 2727 2693 2606 2606
query98 243 233 232 232
query99 1104 1116 1006 1006
Total cold run time: 253224 ms
Total hot run time: 169737 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4538574983
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4477255371
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4494055116
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4538749756
<details>
<summary>TPC-H: <b>Total hot run time: 31296 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2147a5ba9b189ebbbe472cb89b46f3e62357f12, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17676 4034 4045 4034
q2 q3 10807 1390 819 819
q4 4682 475 352 352
q5 7546 2268 2128 2128
q6 268 178 138 138
q7 987 801 635 635
q8 9350 1742 1578 1578
q9 6851 4965 4996 4965
q10 6455 2224 1870 1870
q11 437 276 256 256
q12 699 428 293 293
q13 18227 3401 2768 2768
q14 281 260 238 238
q15 q16 841 786 716 716
q17 1043 922 943 922
q18 7063 5866 5633 5633
q19 1450 1284 1063 1063
q20 510 415 257 257
q21 5623 2619 2321 2321
q22 424 357 310 310
Total cold run time: 101220 ms
Total hot run time: 31296 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4444 4372 4370 4370
q2 q3 4626 4994 4384 4384
q4 2136 2254 1443 1443
q5 4503 4363 4908 4363
q6 260 203 152 152
q7 2004 1900 1613 1613
q8 2545 2167 2460 2167
q9 8114 8097 8016 8016
q10 4817 4996 4471 4471
q11 618 421 388 388
q12 782 787 539 539
q13 3263 3629 2931 2931
q14 295 327 280 280
q15 q16 744 722 646 646
q17 1374 1364 1349 1349
q18 8073 7152 6788 6788
q19 1111 1117 1102 1102
q20 2250 2273 1966 1966
q21 5354 4660 4495 4495
q22 512 477 426 426
Total cold run time: 57825 ms
Total hot run time: 51889 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4527083579
# FE UT Coverage Report
Increment line coverage ` 85.53% (65/76)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_aa63bde8dfa91234456fdd7a5b284c9617eec400/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_aa63bde8dfa91234456fdd7a5b284c9617eec400/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4504575985
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4479585162
# FE UT Coverage Report
Increment line coverage ` 86.90% (73/84)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_8ba0872059324be0298b91f6e90e9fddb3541f06/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_8ba0872059324be0298b91f6e90e9fddb3541f06/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4527001091
<details>
<summary>TPC-DS: <b>Total hot run time: 171823 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa63bde8dfa91234456fdd7a5b284c9617eec400, data reload: false
query5 4337 664 546 546
query6 328 228 197 197
query7 4295 570 292 292
query8 328 252 214 214
query9 8836 4189 4069 4069
query10 445 341 295 295
query11 5838 2428 2271 2271
query12 180 127 121 121
query13 1276 629 429 429
query14 6128 5475 5175 5175
query14_1 4527 4532 4561 4532
query15 213 204 181 181
query16 1027 466 456 456
query17 1129 750 578 578
query18 2478 474 344 344
query19 235 204 155 155
query20 134 132 129 129
query21 207 134 115 115
query22 13678 13648 13374 13374
query23 17312 16485 16215 16215
query23_1 16414 16370 16426 16370
query24 7430 1765 1328 1328
query24_1 1306 1331 1328 1328
query25 561 469 417 417
query26 1362 329 176 176
query27 2647 546 353 353
query28 4495 2037 2026 2026
query29 976 620 493 493
query30 303 240 203 203
query31 1173 1077 950 950
query32 89 72 72 72
query33 533 361 315 315
query34 1192 1142 669 669
query35 771 792 693 693
query36 1434 1403 1295 1295
query37 158 104 94 94
query38 3190 3166 3061 3061
query39 917 899 904 899
query39_1 859 900 876 876
query40 229 159 128 128
query41 67 63 64 63
query42 115 111 108 108
query43 329 331 286 286
query44
query45 211 207 193 193
query46 1086 1180 756 756
query47 2387 2344 2245 2245
query48 391 418 310 310
query49 648 506 389 389
query50 1070 374 261 261
query51 4413 4285 4356 4285
query52 109 112 98 98
query53 262 293 215 215
query54 341 299 268 268
query55 97 94 89 89
query56 316 321 328 321
query57 1436 1440 1365 1365
query58 319 283 293 283
query59 1652 1724 1506 1506
query60 346 347 329 329
query61 188 180 181 180
query62 720 673 591 591
query63 260 218 216 216
query64 2461 863 689 689
query65
query66 1742 496 381 381
query67 30280 30026 29935 29935
query68
query69 457 324 302 302
query70 1021 1004 963 963
query71 317 282 275 275
query72 2984 2676 2346 2346
query73 870 733 446 446
query74 5080 5008 4800 4800
query75 2704 2618 2260 2260
query76 2296 1175 791 791
query77 409 404 342 342
query78 12374 12207 11836 11836
query79 1432 1106 733 733
query80 648 551 471 471
query81 463 278 251 251
query82 1394 159 121 121
query83 359 282 253 253
query84 258 146 115 115
query85 875 557 457 457
query86 393 338 312 312
query87 3403 3399 3279 3279
query88 3635 2733 2742 2733
query89 435 396 348 348
query90 1962 176 182 176
query91 179 168 141 141
query92 75 80 71 71
query93 1562 1463 819 819
query94 546 343 310 310
query95 677 470 348 348
query96 1060 780 333 333
query97 2731 2721 2615 2615
query98 230 228 225 225
query99 1179 1168 1034 1034
Total cold run time: 255091 ms
Total hot run time: 171823 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4527260188
# FE Regression Coverage Report
Increment line coverage ` 88.16% (67/76)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/63318_aa63bde8dfa91234456fdd7a5b284c9617eec400_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/63318_aa63bde8dfa91234456fdd7a5b284c9617eec400_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4478121116
<details>
<summary>TPC-DS: <b>Total hot run time: 170264 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ba0872059324be0298b91f6e90e9fddb3541f06, data reload: false
query5 4320 658 517 517
query6 343 224 204 204
query7 4329 547 313 313
query8 340 236 220 220
query9 8831 3976 3950 3950
query10 454 351 318 318
query11 5780 2350 2174 2174
query12 176 132 127 127
query13 1288 635 435 435
query14 6052 5373 5052 5052
query14_1 4370 4426 4337 4337
query15 214 206 190 190
query16 1046 467 466 466
query17 1176 733 627 627
query18 2558 492 359 359
query19 221 217 163 163
query20 145 135 133 133
query21 226 138 119 119
query22 13769 13572 13327 13327
query23 17224 16350 16038 16038
query23_1 16210 16239 16280 16239
query24 7451 1760 1288 1288
query24_1 1322 1295 1319 1295
query25 592 502 452 452
query26 1318 325 187 187
query27 2702 591 346 346
query28 4413 1957 1998 1957
query29 1044 639 523 523
query30 317 243 201 201
query31 1125 1071 947 947
query32 98 82 83 82
query33 543 364 318 318
query34 1192 1145 688 688
query35 773 772 677 677
query36 1316 1366 1160 1160
query37 153 105 87 87
query38 3208 3192 3061 3061
query39 923 926 880 880
query39_1 886 894 887 887
query40 232 147 126 126
query41 67 63 62 62
query42 113 109 106 106
query43 326 324 286 286
query44
query45 212 199 197 197
query46 1070 1215 730 730
query47 2352 2421 2209 2209
query48 400 423 314 314
query49 640 503 391 391
query50 990 368 251 251
query51 4345 4299 4169 4169
query52 104 111 96 96
query53 258 273 201 201
query54 315 269 271 269
query55 92 95 87 87
query56 307 309 297 297
query57 1434 1387 1296 1296
query58 298 279 268 268
query59 1551 1615 1420 1420
query60 325 323 365 323
query61 161 159 159 159
query62 685 620 562 562
query63 246 201 202 201
query64 2464 826 636 636
query65
query66 1689 483 354 354
query67 30056 29961 29807 29807
query68
query69 458 341 310 310
query70 944 1002 998 998
query71 303 284 276 276
query72 2990 2722 2464 2464
query73 846 749 418 418
query74 5095 4935 4748 4748
query75 2695 2587 2249 2249
query76 2295 1152 771 771
query77 395 414 341 341
query78 12329 12189 11675 11675
query79 1461 1061 774 774
query80 904 533 452 452
query81 505 278 244 244
query82 1373 166 125 125
query83 355 271 253 253
query84 257 141 111 111
query85 916 535 477 477
query86 433 337 335 335
query87 3436 3369 3180 3180
query88 3568 2694 2634 2634
query89 457 384 338 338
query90 1785 187 183 183
query91 179 170 142 142
query92 77 78 72 72
query93 1476 1445 839 839
query94 610 339 337 337
query95 700 462 365 365
query96 1021 777 354 354
query97 2696 2679 2539 2539
query98 248 231 236 231
query99 1110 1178 990 990
Total cold run time: 254166 ms
Total hot run time: 170264 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) optimize InferAggNotNull rule [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4476110260
<details>
<summary>TPC-H: <b>Total hot run time: 31374 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5e656119a9407c4c26e6d0479a04583ae050412b, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17603 3909 3835 3835
q2 q3 10819 1413 813 813
q4 4689 479 341 341
q5 7649 2343 2123 2123
q6 240 175 138 138
q7 929 789 638 638
q8 9353 1745 1657 1657
q9 5154 4888 4945 4888
q10 6394 2103 1801 1801
q11 426 268 244 244
q12 642 419 294 294
q13 18156 3441 2792 2792
q14 271 259 229 229
q15 q16 819 777 712 712
q17 942 977 940 940
q18 6799 5776 5659 5659
q19 1309 1330 1105 1105
q20 529 518 315 315
q21 6179 2874 2535 2535
q22 462 385 315 315
Total cold run time: 99364 ms
Total hot run time: 31374 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4828 4571 4528 4528
q2 q3 4916 5261 4574 4574
q4 2115 2200 1396 1396
q5 5016 4648 4619 4619
q6 250 176 125 125
q7 1909 1734 1511 1511
q8 2392 2034 2061 2034
q9 7682 7221 7167 7167
q10 4487 4396 4002 4002
q11 528 378 351 351
q12 712 735 516 516
q13 3014 3342 2827 2827
q14 263 267 258 258
q15 q16 674 703 607 607
q17 1259 1230 1222 1222
q18 7105 6758 6974 6758
q19 1095 1100 1087 1087
q20 2190 2196 1920 1920
q21 5304 4600 4458 4458
q22 516 463 414 414
Total cold run time: 56255 ms
Total hot run time: 50374 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](nereids) optimize InferAggNotNull rule [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4475157860
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #63318:
URL: https://github.com/apache/doris/pull/63318#discussion_r3273207874
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNull.java:
##########
@@ -47,19 +49,10 @@ public class InferAggNotNull extends OneRewriteRuleFactory {
public Rule build() {
return logicalAggregate()
.when(agg -> agg.getGroupByExpressions().size() == 0)
- .when(agg -> agg.getAggregateFunctions().size() == 1)
- .when(agg -> {
- Set<AggregateFunction> funcs = agg.getAggregateFunctions();
- return funcs.stream().allMatch(f -> f instanceof Count)
- || funcs.stream().allMatch(f -> f instanceof Avg)
- || funcs.stream().allMatch(f -> f instanceof Sum)
- || funcs.stream().allMatch(f -> f instanceof Max)
- || funcs.stream().allMatch(f -> f instanceof Min);
- }).thenApply(ctx -> {
+ .thenApply(ctx -> {
LogicalAggregate<Plan> agg = ctx.root;
- Set<Expression> exprs = agg.getAggregateFunctions().stream().flatMap(f -> f.children().stream())
Review Comment:
the better way is optimize `getAggregateFunctions` directly
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferAggNotNull.java:
##########
@@ -47,19 +49,10 @@ public class InferAggNotNull extends OneRewriteRuleFactory {
public Rule build() {
return logicalAggregate()
.when(agg -> agg.getGroupByExpressions().size() == 0)
- .when(agg -> agg.getAggregateFunctions().size() == 1)
- .when(agg -> {
- Set<AggregateFunction> funcs = agg.getAggregateFunctions();
- return funcs.stream().allMatch(f -> f instanceof Count)
- || funcs.stream().allMatch(f -> f instanceof Avg)
- || funcs.stream().allMatch(f -> f instanceof Sum)
- || funcs.stream().allMatch(f -> f instanceof Max)
- || funcs.stream().allMatch(f -> f instanceof Min);
- }).thenApply(ctx -> {
+ .thenApply(ctx -> {
LogicalAggregate<Plan> agg = ctx.root;
- Set<Expression> exprs = agg.getAggregateFunctions().stream().flatMap(f -> f.children().stream())
- .collect(Collectors.toSet());
- Set<Expression> isNotNulls = ExpressionUtils.inferNotNull(exprs, ctx.cascadesContext);
+ Set<AggregateFunction> aggregateFunctions = collectAggregateFunctions(agg);
+ Set<Expression> isNotNulls = inferCommonNotNulls(aggregateFunctions, ctx.cascadesContext);
Review Comment:
Since ExpressionUtils.inferNotNull has been updated, is it still necessary to use inferCommonNotNulls?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4542315418
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](fe) Bound not-null inference cost [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #63318:
URL: https://github.com/apache/doris/pull/63318#issuecomment-4540835272
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org