You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "yujun777 (via GitHub)" <gi...@apache.org> on 2025/04/11 03:33:51 UTC
[PR] [draft](nereids) Add constant propagation [doris]
yujun777 opened a new pull request, #49982:
URL: https://github.com/apache/doris/pull/49982
### What problem does this PR solve?
support constant propagation, for example: `a = 10 and a + b > 30` will rewrite to `a = 10 and 10 + b > 30`,
when processing a plan, it will collect all its children's equal sets and constants uniforms,
then use them and the plan's expressions to infer more equal sets and constants uniforms,
finally use the combine uniforms to replace this plan's expression's slot with literals.
TODO:
1) process with LogicalRepeat;
2) add more test, include agg, join, subquery, and logical repeat;
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833362580
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833411491
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2811754242
<details>
<summary>TPC-DS: <b>Total hot run time: 191721 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d87aec37767855cd4ec3c2a005b87064810a0710, data reload: false
query1 1413 1096 1074 1074
query2 6082 1813 1792 1792
query3 11018 4479 4693 4479
query4 25964 23944 23134 23134
query5 4844 620 485 485
query6 317 217 216 216
query7 3990 488 279 279
query8 322 247 242 242
query9 8555 2571 2591 2571
query10 490 324 260 260
query11 15918 15101 14844 14844
query12 165 112 112 112
query13 1562 521 394 394
query14 9254 6421 6264 6264
query15 207 208 178 178
query16 7628 631 495 495
query17 1091 780 603 603
query18 2069 423 309 309
query19 205 221 170 170
query20 122 126 120 120
query21 215 132 115 115
query22 4446 4655 4519 4519
query23 34811 33660 33498 33498
query24 8748 2445 2415 2415
query25 521 481 407 407
query26 1226 272 159 159
query27 2806 516 339 339
query28 4801 2184 2157 2157
query29 709 594 446 446
query30 284 228 198 198
query31 900 853 744 744
query32 77 68 69 68
query33 557 373 366 366
query34 799 880 523 523
query35 813 864 783 783
query36 963 1007 924 924
query37 115 107 78 78
query38 4241 4179 4128 4128
query39 1509 1431 1421 1421
query40 212 127 109 109
query41 55 59 57 57
query42 128 111 113 111
query43 509 503 485 485
query44 1319 821 834 821
query45 193 178 170 170
query46 844 1052 641 641
query47 1849 1885 1775 1775
query48 389 415 306 306
query49 798 557 431 431
query50 681 694 410 410
query51 4220 4323 4181 4181
query52 110 105 100 100
query53 229 262 187 187
query54 587 574 519 519
query55 82 83 84 83
query56 320 302 283 283
query57 1188 1180 1127 1127
query58 277 271 267 267
query59 2755 2804 2607 2607
query60 336 329 333 329
query61 131 130 130 130
query62 799 746 685 685
query63 229 196 198 196
query64 4183 1071 703 703
query65 4543 4339 4434 4339
query66 1080 416 302 302
query67 16010 15382 15465 15382
query68 8902 885 509 509
query69 484 309 269 269
query70 1200 1146 1120 1120
query71 468 328 312 312
query72 5657 4797 4845 4797
query73 734 674 355 355
query74 9117 9169 8699 8699
query75 3976 3199 2694 2694
query76 3711 1185 743 743
query77 795 382 283 283
query78 8997 9253 8385 8385
query79 2123 858 578 578
query80 590 517 449 449
query81 490 260 226 226
query82 190 128 103 103
query83 256 244 239 239
query84 241 109 87 87
query85 714 334 269 269
query86 372 271 282 271
query87 4448 4400 4307 4307
query88 3772 2275 2204 2204
query89 394 321 295 295
query90 1901 226 222 222
query91 141 143 112 112
query92 75 63 55 55
query93 1641 995 576 576
query94 609 406 308 308
query95 377 291 292 291
query96 493 564 277 277
query97 3107 3210 3140 3140
query98 243 207 207 207
query99 1306 1433 1294 1294
Total cold run time: 281337 ms
Total hot run time: 191721 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823054829
<details>
<summary>TPC-DS: <b>Total hot run time: 184519 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb7fcae3e6afa2a5aa88d0278ff4850df770d248, data reload: false
query1 994 495 520 495
query2 6555 1839 1866 1839
query3 6746 230 211 211
query4 26328 23232 23276 23232
query5 4377 644 468 468
query6 306 214 200 200
query7 4623 484 273 273
query8 309 268 253 253
query9 8631 2535 2549 2535
query10 484 308 257 257
query11 15333 15158 14888 14888
query12 152 111 110 110
query13 1638 525 387 387
query14 8705 6093 6019 6019
query15 200 190 175 175
query16 7142 634 453 453
query17 941 715 585 585
query18 1980 412 297 297
query19 191 184 160 160
query20 118 124 125 124
query21 213 138 111 111
query22 4023 4079 4135 4079
query23 33861 32854 32966 32854
query24 8491 2353 2386 2353
query25 553 470 425 425
query26 1279 262 147 147
query27 2765 491 325 325
query28 4331 2087 2067 2067
query29 751 535 422 422
query30 283 213 187 187
query31 868 790 724 724
query32 72 64 68 64
query33 561 350 315 315
query34 795 838 517 517
query35 799 809 754 754
query36 978 982 878 878
query37 113 101 75 75
query38 4090 4127 4144 4127
query39 1463 1386 1408 1386
query40 208 121 108 108
query41 57 57 57 57
query42 120 108 111 108
query43 489 508 472 472
query44 1271 799 797 797
query45 182 179 162 162
query46 805 1016 605 605
query47 1753 1774 1707 1707
query48 382 405 308 308
query49 815 494 417 417
query50 642 675 401 401
query51 4197 4115 4028 4028
query52 102 106 95 95
query53 225 263 183 183
query54 568 554 500 500
query55 82 81 86 81
query56 317 321 300 300
query57 1099 1165 1108 1108
query58 266 254 272 254
query59 2616 2705 2622 2622
query60 322 350 300 300
query61 129 126 127 126
query62 753 731 683 683
query63 221 187 187 187
query64 4358 1006 677 677
query65 4309 4236 4238 4236
query66 1140 402 305 305
query67 15816 15296 15334 15296
query68 7904 875 503 503
query69 500 300 275 275
query70 1195 1115 1123 1115
query71 450 314 292 292
query72 5424 4635 4635 4635
query73 675 555 335 335
query74 8813 8902 8765 8765
query75 3735 3237 2676 2676
query76 3624 1187 758 758
query77 800 371 282 282
query78 9365 9350 8548 8548
query79 2209 864 563 563
query80 579 511 450 450
query81 482 253 220 220
query82 477 125 100 100
query83 258 245 229 229
query84 297 102 84 84
query85 779 344 353 344
query86 391 315 289 289
query87 4376 4462 4300 4300
query88 3543 2236 2238 2236
query89 380 313 279 279
query90 1850 208 213 208
query91 138 140 110 110
query92 81 63 58 58
query93 1668 964 563 563
query94 686 390 317 317
query95 373 283 289 283
query96 476 566 274 274
query97 3150 3250 3060 3060
query98 242 213 203 203
query99 1388 1398 1271 1271
Total cold run time: 271594 ms
Total hot run time: 184519 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2827565682
<details>
<summary>TPC-DS: <b>Total hot run time: 192055 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bac9e2e46a55da229928c9bdb42e58fb56761025, data reload: false
query1 1438 1068 1055 1055
query2 6253 1817 1797 1797
query3 11128 4646 4580 4580
query4 53762 25488 23061 23061
query5 5268 533 481 481
query6 350 214 198 198
query7 4927 483 288 288
query8 327 278 244 244
query9 5838 2610 2627 2610
query10 465 330 314 314
query11 15168 15033 14897 14897
query12 155 114 104 104
query13 1073 489 379 379
query14 10108 6341 6504 6341
query15 208 196 183 183
query16 7136 675 518 518
query17 1087 773 605 605
query18 1547 410 321 321
query19 207 201 167 167
query20 128 133 124 124
query21 209 143 112 112
query22 4331 4485 4298 4298
query23 34250 33215 33614 33215
query24 6738 2399 2457 2399
query25 463 483 415 415
query26 685 278 153 153
query27 2171 504 336 336
query28 3312 2189 2158 2158
query29 567 576 452 452
query30 267 223 245 223
query31 828 816 766 766
query32 83 72 64 64
query33 451 362 324 324
query34 757 844 537 537
query35 812 854 761 761
query36 943 999 875 875
query37 116 105 80 80
query38 4225 4302 4177 4177
query39 1505 1477 1451 1451
query40 222 123 108 108
query41 60 58 56 56
query42 124 108 107 107
query43 500 487 495 487
query44 1331 838 832 832
query45 183 182 175 175
query46 855 1042 653 653
query47 1849 1869 1778 1778
query48 404 405 301 301
query49 699 535 462 462
query50 664 701 413 413
query51 4198 4272 4125 4125
query52 112 106 95 95
query53 226 268 201 201
query54 597 583 536 536
query55 81 90 86 86
query56 326 307 304 304
query57 1169 1252 1132 1132
query58 286 272 264 264
query59 2774 2798 2710 2710
query60 340 346 312 312
query61 134 152 141 141
query62 728 769 668 668
query63 228 185 189 185
query64 1562 1031 700 700
query65 4521 4234 4240 4234
query66 725 392 300 300
query67 15915 15589 15620 15589
query68 7356 880 518 518
query69 539 304 264 264
query70 1222 1135 1069 1069
query71 521 304 296 296
query72 5665 4844 5043 4844
query73 1427 712 348 348
query74 9001 9190 8684 8684
query75 3820 3217 2825 2825
query76 4337 1186 752 752
query77 623 373 282 282
query78 9322 9508 8651 8651
query79 1935 854 562 562
query80 644 549 438 438
query81 490 253 219 219
query82 436 134 102 102
query83 363 258 234 234
query84 289 102 89 89
query85 818 327 287 287
query86 372 299 277 277
query87 4483 4442 4424 4424
query88 3224 2205 2207 2205
query89 411 314 283 283
query90 1949 203 206 203
query91 141 146 108 108
query92 78 62 58 58
query93 1279 943 586 586
query94 669 382 303 303
query95 367 295 287 287
query96 472 564 268 268
query97 3189 3248 3102 3102
query98 231 211 201 201
query99 1399 1424 1269 1269
Total cold run time: 297171 ms
Total hot run time: 192055 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823932014
<details>
<summary>TPC-DS: <b>Total hot run time: 185142 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 380ba6cacf44785c79fbf8a3313cb0f6d054a66e, data reload: false
query1 1048 479 478 478
query2 6570 1797 1779 1779
query3 6745 229 220 220
query4 25989 23746 23411 23411
query5 4365 627 469 469
query6 298 203 213 203
query7 4628 480 281 281
query8 282 233 241 233
query9 8605 2537 2546 2537
query10 441 333 253 253
query11 15738 15112 14800 14800
query12 177 112 105 105
query13 1649 502 396 396
query14 8673 6087 6170 6087
query15 202 189 169 169
query16 7128 653 455 455
query17 1020 694 562 562
query18 1974 400 300 300
query19 183 185 157 157
query20 117 115 120 115
query21 218 128 105 105
query22 4003 4331 4006 4006
query23 33928 33030 33047 33030
query24 8447 2358 2366 2358
query25 555 454 389 389
query26 1225 264 153 153
query27 2779 536 332 332
query28 4368 2100 2078 2078
query29 769 549 433 433
query30 282 214 191 191
query31 879 810 741 741
query32 74 67 68 67
query33 556 376 308 308
query34 787 860 507 507
query35 766 816 731 731
query36 954 964 898 898
query37 110 98 77 77
query38 4159 4181 4088 4088
query39 1461 1375 1413 1375
query40 217 129 112 112
query41 64 61 63 61
query42 127 119 109 109
query43 494 506 478 478
query44 1305 840 796 796
query45 181 177 166 166
query46 838 1033 615 615
query47 1707 1773 1738 1738
query48 370 408 314 314
query49 796 530 429 429
query50 651 688 389 389
query51 4121 4134 4073 4073
query52 104 106 100 100
query53 231 286 181 181
query54 574 574 502 502
query55 83 82 78 78
query56 308 312 294 294
query57 1127 1146 1055 1055
query58 270 261 257 257
query59 2598 2605 2611 2605
query60 322 325 304 304
query61 140 129 144 129
query62 791 711 671 671
query63 233 187 187 187
query64 4365 1037 754 754
query65 4360 4229 4258 4229
query66 1169 427 321 321
query67 15718 15561 15722 15561
query68 7532 881 530 530
query69 496 314 285 285
query70 1186 1135 1108 1108
query71 416 319 304 304
query72 6016 4750 4623 4623
query73 634 584 341 341
query74 9228 9129 8699 8699
query75 3210 3211 2718 2718
query76 3167 1190 758 758
query77 488 385 290 290
query78 9366 9465 8640 8640
query79 2317 810 558 558
query80 574 523 443 443
query81 508 252 225 225
query82 478 130 102 102
query83 256 249 237 237
query84 301 120 86 86
query85 796 334 282 282
query86 393 295 313 295
query87 4392 4507 4243 4243
query88 3778 2240 2264 2240
query89 387 320 281 281
query90 1882 215 219 215
query91 156 146 107 107
query92 71 64 58 58
query93 1890 992 580 580
query94 697 405 313 313
query95 374 300 289 289
query96 483 566 277 277
query97 3170 3240 3101 3101
query98 238 215 203 203
query99 1350 1403 1326 1326
Total cold run time: 271646 ms
Total hot run time: 185142 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838895817
<details>
<summary>TPC-H: <b>Total hot run time: 33850 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ab930e4985c14df6decf4c7d449aea9ef0685c2, data reload: false
------ Round 1 ----------------------------------
q1 25767 5044 4992 4992
q2 2071 310 185 185
q3 10370 1251 674 674
q4 10217 972 513 513
q5 8007 2412 2341 2341
q6 185 163 131 131
q7 917 725 613 613
q8 9305 1265 1086 1086
q9 6732 5174 5138 5138
q10 6848 2308 1891 1891
q11 482 277 278 277
q12 354 351 210 210
q13 17779 3664 3070 3070
q14 224 229 218 218
q15 549 499 493 493
q16 410 422 369 369
q17 608 863 361 361
q18 7523 7230 7072 7072
q19 1588 960 550 550
q20 313 320 224 224
q21 4222 2755 2480 2480
q22 1054 1041 962 962
Total cold run time: 115525 ms
Total hot run time: 33850 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5196 5079 5082 5079
q2 248 354 236 236
q3 2185 2584 2272 2272
q4 1312 1735 1365 1365
q5 4424 4435 4340 4340
q6 222 167 128 128
q7 1975 1933 1779 1779
q8 2580 2623 2614 2614
q9 7238 7287 7071 7071
q10 2972 3168 2719 2719
q11 564 496 502 496
q12 686 737 628 628
q13 3522 3927 3314 3314
q14 279 321 290 290
q15 522 474 472 472
q16 453 475 421 421
q17 1168 1565 1384 1384
q18 7738 7556 7473 7473
q19 826 813 859 813
q20 1982 1977 1842 1842
q21 5029 4744 4701 4701
q22 1086 1082 1043 1043
Total cold run time: 52207 ms
Total hot run time: 50480 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2952734616
<details>
<summary>TPC-DS: <b>Total hot run time: 191832 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9284067656c48e175b43cffa6577d6dfaec077b7, data reload: false
query1 1428 1065 1100 1065
query2 6214 1832 1828 1828
query3 11045 4405 4469 4405
query4 54953 25911 22943 22943
query5 5105 559 471 471
query6 351 215 201 201
query7 4889 490 291 291
query8 295 245 218 218
query9 5778 2627 2631 2627
query10 400 341 270 270
query11 15211 15023 14873 14873
query12 166 110 108 108
query13 1042 540 417 417
query14 10223 6188 6348 6188
query15 206 211 176 176
query16 7079 650 462 462
query17 1070 732 595 595
query18 1543 404 339 339
query19 195 185 162 162
query20 124 120 116 116
query21 209 129 109 109
query22 4354 4555 4309 4309
query23 34228 33557 33596 33557
query24 6569 2427 2420 2420
query25 479 491 425 425
query26 684 278 156 156
query27 2271 516 346 346
query28 3049 2165 2159 2159
query29 619 617 472 472
query30 274 229 195 195
query31 864 829 752 752
query32 76 73 69 69
query33 467 374 355 355
query34 793 878 551 551
query35 826 828 728 728
query36 933 996 920 920
query37 120 108 80 80
query38 4253 4230 4180 4180
query39 1514 1478 1470 1470
query40 215 129 110 110
query41 66 60 56 56
query42 125 119 112 112
query43 510 523 497 497
query44 1335 842 833 833
query45 183 187 171 171
query46 868 1024 660 660
query47 1835 1916 1784 1784
query48 404 443 344 344
query49 675 499 435 435
query50 659 697 403 403
query51 4187 4239 4205 4205
query52 108 107 99 99
query53 236 265 184 184
query54 583 579 513 513
query55 91 86 82 82
query56 295 325 290 290
query57 1184 1219 1136 1136
query58 291 268 279 268
query59 2799 2828 2604 2604
query60 356 342 328 328
query61 133 145 130 130
query62 707 737 692 692
query63 240 190 192 190
query64 1506 1035 686 686
query65 4233 4143 4181 4143
query66 766 405 298 298
query67 16060 15562 15451 15451
query68 7534 888 539 539
query69 554 304 272 272
query70 1262 1121 1074 1074
query71 511 325 304 304
query72 5503 4861 4920 4861
query73 1399 673 359 359
query74 8911 9189 9066 9066
query75 3892 3197 2683 2683
query76 4301 1189 742 742
query77 652 382 282 282
query78 9389 9417 8708 8708
query79 1917 885 566 566
query80 576 516 448 448
query81 477 257 227 227
query82 410 128 98 98
query83 252 254 250 250
query84 292 114 91 91
query85 793 410 322 322
query86 371 304 278 278
query87 4386 4479 4422 4422
query88 3159 2326 2298 2298
query89 401 307 292 292
query90 1970 215 208 208
query91 141 145 111 111
query92 79 59 59 59
query93 1155 928 590 590
query94 684 419 308 308
query95 430 301 287 287
query96 494 581 281 281
query97 2708 2729 2632 2632
query98 233 203 206 203
query99 1440 1416 1259 1259
Total cold run time: 296825 ms
Total hot run time: 191832 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2954000148
<details>
<summary>TPC-H: <b>Total hot run time: 33725 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 494426d754a065a8b4eb6c2a4a7d4ee6133193db, data reload: false
------ Round 1 ----------------------------------
q1 25992 5031 5006 5006
q2 1957 273 171 171
q3 10298 1206 717 717
q4 10594 1049 578 578
q5 7553 2344 2347 2344
q6 179 165 134 134
q7 884 758 636 636
q8 9338 1223 1126 1126
q9 6980 5012 5065 5012
q10 6862 2324 1886 1886
q11 488 284 268 268
q12 340 344 208 208
q13 17794 3695 3085 3085
q14 230 239 217 217
q15 565 485 475 475
q16 420 428 377 377
q17 592 849 355 355
q18 7566 7128 7044 7044
q19 1665 942 540 540
q20 325 317 226 226
q21 3798 2539 2354 2354
q22 1021 1015 966 966
Total cold run time: 115441 ms
Total hot run time: 33725 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5200 5041 5067 5041
q2 238 327 226 226
q3 2146 2638 2263 2263
q4 1339 1784 1378 1378
q5 4396 4317 4492 4317
q6 223 173 130 130
q7 1996 1899 1758 1758
q8 2543 2620 2493 2493
q9 7080 7212 6923 6923
q10 3022 3184 2738 2738
q11 590 528 508 508
q12 650 777 598 598
q13 3534 3887 3260 3260
q14 298 315 270 270
q15 523 476 483 476
q16 458 497 449 449
q17 1138 1498 1363 1363
q18 7716 7489 7332 7332
q19 829 802 842 802
q20 1963 2054 1919 1919
q21 4824 4311 4384 4311
q22 1089 1076 1035 1035
Total cold run time: 51795 ms
Total hot run time: 49590 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2914908332
<details>
<summary>TPC-DS: <b>Total hot run time: 191264 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5de5b4e8038ca6b71c5470481e59e65c6de3bb9f, data reload: false
query1 1415 1106 1080 1080
query2 6291 1778 1770 1770
query3 11156 4758 4604 4604
query4 25579 23376 22948 22948
query5 4075 640 467 467
query6 324 204 205 204
query7 4001 495 285 285
query8 320 257 258 257
query9 8483 2585 2595 2585
query10 488 324 262 262
query11 15424 15100 14763 14763
query12 159 113 118 113
query13 1570 563 405 405
query14 9206 6075 6173 6075
query15 203 186 160 160
query16 7575 629 452 452
query17 1175 696 582 582
query18 2010 424 316 316
query19 201 195 166 166
query20 124 134 122 122
query21 206 136 112 112
query22 4606 4564 4441 4441
query23 35105 33760 33605 33605
query24 8882 2471 2485 2471
query25 499 467 392 392
query26 890 275 150 150
query27 2708 525 337 337
query28 4680 2170 2154 2154
query29 642 581 454 454
query30 272 218 196 196
query31 888 817 732 732
query32 83 63 64 63
query33 547 393 324 324
query34 809 868 548 548
query35 803 820 737 737
query36 979 990 890 890
query37 114 95 77 77
query38 4253 4260 4187 4187
query39 1570 1476 1481 1476
query40 213 125 118 118
query41 73 67 71 67
query42 125 127 122 122
query43 505 512 482 482
query44 1341 868 849 849
query45 186 186 176 176
query46 862 1028 660 660
query47 1831 1890 1823 1823
query48 418 463 327 327
query49 759 493 445 445
query50 670 691 422 422
query51 4171 4197 4217 4197
query52 108 108 100 100
query53 236 266 191 191
query54 588 593 525 525
query55 86 89 85 85
query56 300 318 303 303
query57 1214 1205 1124 1124
query58 272 262 260 260
query59 2594 2734 2616 2616
query60 345 344 324 324
query61 133 126 142 126
query62 787 737 681 681
query63 220 183 194 183
query64 3365 1053 684 684
query65 4413 4365 4349 4349
query66 849 416 306 306
query67 15973 15713 15412 15412
query68 8845 884 515 515
query69 486 309 263 263
query70 1278 1099 1089 1089
query71 481 325 307 307
query72 5224 4676 4729 4676
query73 705 584 362 362
query74 9194 9028 8738 8738
query75 4113 3156 2697 2697
query76 3648 1224 735 735
query77 771 391 292 292
query78 9396 9512 8703 8703
query79 2069 831 578 578
query80 598 518 453 453
query81 498 254 215 215
query82 447 124 92 92
query83 308 255 231 231
query84 249 109 85 85
query85 764 327 285 285
query86 377 302 301 301
query87 4332 4398 4257 4257
query88 3440 2264 2254 2254
query89 398 314 279 279
query90 1850 205 211 205
query91 143 144 110 110
query92 73 60 58 58
query93 1677 962 571 571
query94 666 418 318 318
query95 370 288 284 284
query96 494 584 287 287
query97 2707 2760 2622 2622
query98 236 205 204 204
query99 1403 1354 1279 1279
Total cold run time: 278192 ms
Total hot run time: 191264 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2879244371
<details>
<summary>ClickBench: <b>Total hot run time: 29.83 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c17efe382987bbb5fe265707273ae1c879a3a35, data reload: false
query1 0.05 0.04 0.03
query2 0.12 0.10 0.11
query3 0.24 0.20 0.19
query4 1.59 0.18 0.19
query5 0.59 0.59 0.58
query6 1.20 0.73 0.73
query7 0.02 0.01 0.02
query8 0.04 0.03 0.04
query9 0.58 0.52 0.53
query10 0.57 0.57 0.57
query11 0.16 0.12 0.11
query12 0.15 0.11 0.12
query13 0.61 0.60 0.59
query14 0.79 0.80 0.81
query15 0.89 0.86 0.88
query16 0.41 0.37 0.38
query17 1.08 1.06 1.04
query18 0.24 0.22 0.22
query19 1.95 1.81 1.91
query20 0.02 0.02 0.01
query21 15.40 0.93 0.57
query22 0.78 1.30 0.68
query23 14.79 1.37 0.60
query24 6.93 1.17 1.05
query25 0.44 0.13 0.14
query26 0.69 0.18 0.14
query27 0.05 0.05 0.06
query28 9.29 0.93 0.47
query29 12.62 4.06 3.37
query30 0.25 0.10 0.07
query31 2.82 0.59 0.39
query32 3.23 0.54 0.46
query33 3.03 3.11 3.13
query34 15.74 5.07 4.53
query35 4.53 4.53 4.55
query36 0.67 0.50 0.48
query37 0.09 0.08 0.06
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.18 0.15 0.13
query41 0.08 0.02 0.02
query42 0.03 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 103.07 s
Total hot run time: 29.83 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2898498423
<details>
<summary>TPC-H: <b>Total hot run time: 35262 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8398b2cd745248ad60a0a654105ed3b53b02bf7b, data reload: false
------ Round 1 ----------------------------------
q1 25823 5294 5210 5210
q2 2104 280 177 177
q3 10445 1292 693 693
q4 10239 1041 539 539
q5 7535 2459 2443 2443
q6 202 165 134 134
q7 969 761 625 625
q8 9318 1487 1215 1215
q9 7884 5257 5304 5257
q10 6868 2375 1929 1929
q11 496 305 285 285
q12 355 374 223 223
q13 18443 3840 3294 3294
q14 237 234 215 215
q15 540 505 491 491
q16 452 458 404 404
q17 630 916 389 389
q18 7870 7837 7588 7588
q19 3890 1043 555 555
q20 338 348 222 222
q21 3964 3296 2430 2430
q22 1063 1011 944 944
Total cold run time: 119665 ms
Total hot run time: 35262 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5568 5347 5493 5347
q2 255 326 234 234
q3 2218 2810 2470 2470
q4 1469 1872 1462 1462
q5 4520 4377 4356 4356
q6 236 170 137 137
q7 1986 1966 1765 1765
q8 2754 2604 2637 2604
q9 7045 7110 7063 7063
q10 3082 3193 2745 2745
q11 589 516 497 497
q12 707 790 595 595
q13 3582 4047 3345 3345
q14 274 305 269 269
q15 531 479 480 479
q16 446 491 449 449
q17 1184 1599 1430 1430
q18 7842 7683 7304 7304
q19 834 835 1155 835
q20 2038 2072 1901 1901
q21 5097 4503 4402 4402
q22 1094 1085 1033 1033
Total cold run time: 53351 ms
Total hot run time: 50722 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890427918
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890673207
<details>
<summary>ClickBench: <b>Total hot run time: 29 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ee31a1ae3d3e603ceadb2c81ce12e1aec375f8da, data reload: false
query1 0.04 0.04 0.04
query2 0.12 0.10 0.13
query3 0.26 0.19 0.19
query4 1.60 0.19 0.11
query5 0.43 0.41 0.43
query6 1.15 0.66 0.66
query7 0.03 0.01 0.02
query8 0.04 0.04 0.04
query9 0.59 0.51 0.52
query10 0.56 0.59 0.56
query11 0.15 0.10 0.11
query12 0.14 0.12 0.12
query13 0.62 0.60 0.60
query14 0.78 0.80 0.82
query15 0.90 0.85 0.83
query16 0.38 0.39 0.39
query17 1.03 1.09 1.02
query18 0.22 0.20 0.21
query19 1.88 1.85 1.78
query20 0.02 0.01 0.02
query21 15.39 0.94 0.55
query22 0.76 1.11 0.65
query23 15.43 1.38 0.65
query24 7.38 1.28 0.97
query25 0.47 0.19 0.07
query26 0.63 0.16 0.14
query27 0.05 0.05 0.05
query28 9.52 0.90 0.45
query29 12.54 3.97 3.28
query30 0.24 0.09 0.08
query31 2.84 0.60 0.38
query32 3.24 0.56 0.46
query33 3.10 3.00 3.11
query34 15.80 5.12 4.52
query35 4.50 4.49 4.50
query36 0.67 0.50 0.47
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.02 0.02 0.02
query40 0.18 0.15 0.14
query41 0.08 0.04 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 103.99 s
Total hot run time: 29 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2893889561
<details>
<summary>ClickBench: <b>Total hot run time: 28.75 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 077b5b29596ba046f9e36350c68df03dd6caf0aa, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.10 0.11
query3 0.26 0.20 0.19
query4 1.61 0.20 0.20
query5 0.44 0.42 0.42
query6 1.18 0.68 0.66
query7 0.03 0.01 0.02
query8 0.04 0.03 0.03
query9 0.59 0.54 0.51
query10 0.57 0.57 0.58
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.62 0.59 0.61
query14 0.79 0.80 0.81
query15 0.90 0.86 0.89
query16 0.37 0.38 0.39
query17 1.02 1.03 1.05
query18 0.22 0.21 0.21
query19 1.96 1.89 1.78
query20 0.02 0.02 0.02
query21 15.43 0.92 0.56
query22 0.77 1.24 0.78
query23 14.74 1.35 0.64
query24 7.74 0.87 0.34
query25 0.50 0.07 0.06
query26 0.60 0.18 0.14
query27 0.05 0.05 0.05
query28 9.47 0.89 0.45
query29 12.57 3.92 3.28
query30 0.26 0.09 0.07
query31 2.82 0.61 0.39
query32 3.23 0.55 0.48
query33 2.98 3.04 3.09
query34 15.73 5.12 4.49
query35 4.59 4.58 4.56
query36 0.68 0.49 0.48
query37 0.09 0.07 0.06
query38 0.05 0.04 0.03
query39 0.04 0.02 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 103.78 s
Total hot run time: 28.75 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209087832
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -210,7 +217,54 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
Review Comment:
udpate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076612885
<details>
<summary>TPC-DS: <b>Total hot run time: 183357 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79a5db32079ea3bcb6f30e554e2924fe379da810, data reload: false
query1 1001 404 383 383
query2 6513 1692 1655 1655
query3 6744 213 211 211
query4 26702 23696 23035 23035
query5 4334 582 475 475
query6 304 211 203 203
query7 4626 491 288 288
query8 262 224 205 205
query9 8614 2655 2669 2655
query10 484 329 283 283
query11 15652 15152 14951 14951
query12 156 109 108 108
query13 1662 538 398 398
query14 8856 5806 5802 5802
query15 200 179 171 171
query16 7362 622 462 462
query17 1181 699 561 561
query18 1969 400 290 290
query19 190 185 151 151
query20 119 122 114 114
query21 213 117 101 101
query22 4176 4399 4190 4190
query23 34062 33113 33255 33113
query24 8454 2387 2338 2338
query25 553 474 387 387
query26 1244 266 150 150
query27 2769 501 340 340
query28 4315 2154 2130 2130
query29 771 575 456 456
query30 285 223 190 190
query31 877 820 726 726
query32 75 69 63 63
query33 573 380 343 343
query34 827 865 515 515
query35 784 818 749 749
query36 963 1027 908 908
query37 112 101 79 79
query38 4151 4145 4055 4055
query39 1496 1420 1428 1420
query40 210 115 104 104
query41 57 53 53 53
query42 124 103 103 103
query43 531 511 488 488
query44 1352 828 824 824
query45 176 172 167 167
query46 835 1012 610 610
query47 1758 1844 1724 1724
query48 385 435 303 303
query49 725 463 371 371
query50 632 684 408 408
query51 5412 5550 5407 5407
query52 105 109 93 93
query53 220 243 184 184
query54 557 588 488 488
query55 82 79 78 78
query56 297 294 271 271
query57 1185 1212 1139 1139
query58 267 262 256 256
query59 2602 2709 2559 2559
query60 325 334 314 314
query61 128 122 121 121
query62 802 717 660 660
query63 228 182 181 181
query64 4311 1022 680 680
query65 4295 4211 4186 4186
query66 1140 415 325 325
query67 15942 15728 15391 15391
query68 8528 895 535 535
query69 488 313 270 270
query70 1249 1170 1092 1092
query71 464 321 298 298
query72 5385 2355 5058 2355
query73 781 753 354 354
query74 9165 8932 8796 8796
query75 3924 3209 2680 2680
query76 3633 1139 710 710
query77 812 374 306 306
query78 9227 9482 8729 8729
query79 1148 890 571 571
query80 570 506 452 452
query81 467 253 232 232
query82 294 132 97 97
query83 269 248 246 246
query84 250 104 83 83
query85 749 417 317 317
query86 338 312 289 289
query87 4424 4421 4327 4327
query88 2850 2332 2251 2251
query89 372 320 278 278
query90 2055 205 202 202
query91 143 138 117 117
query92 79 63 56 56
query93 1044 955 580 580
query94 673 394 309 309
query95 374 294 280 280
query96 489 576 277 277
query97 2690 2758 2647 2647
query98 227 210 209 209
query99 1431 1400 1270 1270
Total cold run time: 272906 ms
Total hot run time: 183357 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2194806945
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java:
##########
@@ -301,6 +303,8 @@ public class Rewriter extends AbstractBatchJobExecutor {
new InferFilterNotNull(),
new InferJoinNotNull()
),
+ bottomUp(RuleSet.PUSH_DOWN_FILTERS),
Review Comment:
add comment to explain why need add a push down filters here
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java:
##########
@@ -99,11 +100,15 @@ public static ConnectContext createBasicMvContext(@Nullable ConnectContext paren
ctx.getSessionVariable().showHiddenColumns = false;
ctx.getSessionVariable().allowModifyMaterializedViewData = true;
// Disable add default limit rule to avoid refresh data wrong
Review Comment:
should update comment
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/TreeNode.java:
##########
@@ -150,22 +150,20 @@ default NODE_TYPE rewriteDownShortCircuit(Function<NODE_TYPE, NODE_TYPE> rewrite
/**
* similar to rewriteDownShortCircuit, except that only subtrees, whose root satisfies
- * border predicate are rewritten.
+ * border predicate are rewritten, and if a node not match predicate, then its descendant will not rewrite.
*/
default NODE_TYPE rewriteDownShortCircuitDown(Function<NODE_TYPE, NODE_TYPE> rewriteFunction,
- Predicate border, boolean aboveBorder) {
+ Predicate predicate) {
Review Comment:
why modify this infra?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalProject.java:
##########
@@ -121,9 +122,13 @@ public String shapeInfo() {
&& context.getSessionVariable().getDetailShapePlanNodesSet().contains(getClass().getSimpleName())) {
StringBuilder builder = new StringBuilder();
builder.append(getClass().getSimpleName());
- // the internal project list's order may be unstable, especial for join tables,
- // so sort the projects to make it stable
- builder.append(projects.stream().map(Expression::shapeInfo).sorted()
+ Stream<String> projectStream = projects.stream().map(Expression::shapeInfo);
+ if (!context.getSessionVariable().enableProjectShapePositionOrder) {
Review Comment:
maybe we could sorted it all times
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
Review Comment:
use ImmutableSet directly
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
Review Comment:
use for loop for better perfermance
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -210,7 +217,54 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
+ ExpressionRewriteContext context = new ExpressionRewriteContext(tempCascadeContext);
Review Comment:
and furthermore, could `checkConstantExpr` do not relay on constant folding? if it need a literal, we should do it apply enough expression rewrite before use it
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java:
##########
@@ -227,6 +227,13 @@ public Plan visitLogicalUnion(LogicalUnion union, PruneContext context) {
.filter(index -> prunedOutput.contains(originOutput.get(index)))
.boxed()
.collect(ImmutableList.toImmutableList());
+ // in function `pruneUnionOutput`, the prunedOutput may not come from originOutput,
+ // it's just a new add column slot reference.
+ if (prunedOutputIndexes.isEmpty()) {
+ prunedOutputIndexes = IntStream.range(0, prunedOutput.size())
Review Comment:
why add back all?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionNormalizationAndOptimization.java:
##########
@@ -21,17 +21,23 @@
import com.google.common.collect.ImmutableList;
+import java.util.List;
+
/** ExpressionNormalizationAndOptimization */
public class ExpressionNormalizationAndOptimization extends ExpressionRewrite {
/** ExpressionNormalizationAndOptimization */
- public ExpressionNormalizationAndOptimization() {
- super(new ExpressionRuleExecutor(
- ImmutableList.<ExpressionRewriteRule<ExpressionRewriteContext>>builder()
- .addAll(ExpressionNormalization.NORMALIZE_REWRITE_RULES)
- .addAll(ExpressionOptimization.OPTIMIZE_REWRITE_RULES)
- .addAll(ExpressionOptimization.ADD_RANGE)
- .build()
- ));
+ public ExpressionNormalizationAndOptimization(boolean addRange) {
+ super(new ExpressionRuleExecutor(buildRewriteRule(addRange)));
+ }
+
+ private static List<ExpressionRewriteRule<ExpressionRewriteContext>> buildRewriteRule(boolean addRange) {
+ ImmutableList.Builder<ExpressionRewriteRule<ExpressionRewriteContext>> builder = ImmutableList.builder();
+ builder.addAll(ExpressionNormalization.NORMALIZE_REWRITE_RULES)
+ .addAll(ExpressionOptimization.OPTIMIZE_REWRITE_RULES);
+ if (addRange) {
Review Comment:
add comment for this function to explain why need a new parameter `addRange`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -210,7 +217,54 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
Review Comment:
should check ConnectContext.get() == null
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java:
##########
@@ -301,6 +303,8 @@ public class Rewriter extends AbstractBatchJobExecutor {
new InferFilterNotNull(),
new InferJoinNotNull()
),
+ bottomUp(RuleSet.PUSH_DOWN_FILTERS),
+ custom(RuleType.CONSTANT_PROPAGATION, ConstantPropagation::new),
Review Comment:
add comment to explain why need to again here and below
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
Review Comment:
could be static?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -210,7 +217,54 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
+ ExpressionRewriteContext context = new ExpressionRewriteContext(tempCascadeContext);
Review Comment:
if just need to do constant folding, could we not relay on CascadesContext? i saw only few function relay on `CascadesContext`, and them should be folding in the very beginning, so maybe we could pass a `null` to `ExpressionRewriteContext` and let FoldConstantByFE be safe for a null pointer `CascadesContext`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
Review Comment:
we do constant propagation when plan still has apply node?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
Review Comment:
i think when we do constant propagation, the repeat node only do repeat things
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
Review Comment:
if we could rewrite mark conditions to eliminate some slot from subquery, maybe we could remove some join directly @starocean999 please check with @yujun777
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
Review Comment:
maybe u should override it explicitly and let it call `visit()` directly to avoid to override it by mistake in the future
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
Review Comment:
why not false or null? what if a is nullable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3053121326
<details>
<summary>TPC-DS: <b>Total hot run time: 185473 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0ac0a854575f8810270790b6a58154b043b9529c, data reload: false
query1 1011 379 407 379
query2 6508 1749 1729 1729
query3 6759 218 219 218
query4 26038 23752 23516 23516
query5 4435 582 454 454
query6 305 219 208 208
query7 4635 500 298 298
query8 273 237 210 210
query9 8679 2666 2655 2655
query10 451 340 276 276
query11 15524 15110 14909 14909
query12 149 106 102 102
query13 1637 554 401 401
query14 9324 5718 5732 5718
query15 226 192 169 169
query16 7445 637 460 460
query17 1187 700 582 582
query18 1999 411 313 313
query19 200 187 160 160
query20 122 120 120 120
query21 214 126 113 113
query22 4165 4112 4141 4112
query23 34035 33252 33002 33002
query24 8491 2399 2386 2386
query25 525 469 393 393
query26 1234 303 151 151
query27 2723 530 346 346
query28 4343 2120 2114 2114
query29 751 573 456 456
query30 286 223 190 190
query31 908 808 736 736
query32 73 65 66 65
query33 582 362 331 331
query34 788 835 517 517
query35 780 842 742 742
query36 969 984 854 854
query37 125 99 80 80
query38 4299 4117 4099 4099
query39 1487 1424 1425 1424
query40 215 125 111 111
query41 70 61 69 61
query42 121 114 148 114
query43 513 529 503 503
query44 1357 828 825 825
query45 178 175 164 164
query46 838 1014 625 625
query47 1721 1784 1708 1708
query48 383 413 316 316
query49 747 477 424 424
query50 653 687 402 402
query51 4153 4423 4179 4179
query52 116 113 99 99
query53 229 253 193 193
query54 582 569 507 507
query55 88 85 81 81
query56 321 319 303 303
query57 1176 1174 1158 1158
query58 313 264 254 254
query59 2635 2678 2620 2620
query60 329 319 311 311
query61 134 129 131 129
query62 813 712 670 670
query63 229 194 190 190
query64 4269 1030 686 686
query65 4263 4189 4276 4189
query66 1107 408 314 314
query67 15720 15611 15307 15307
query68 6459 882 534 534
query69 481 311 266 266
query70 1187 1111 1097 1097
query71 424 327 310 310
query72 5639 4876 4881 4876
query73 678 661 353 353
query74 9383 9113 8813 8813
query75 3198 3188 2762 2762
query76 3218 1166 741 741
query77 508 403 309 309
query78 9238 9462 8626 8626
query79 994 880 569 569
query80 596 537 475 475
query81 492 270 225 225
query82 177 130 96 96
query83 266 260 254 254
query84 261 113 90 90
query85 831 435 404 404
query86 367 324 309 309
query87 4470 4442 4516 4442
query88 2949 2307 2322 2307
query89 467 314 277 277
query90 1756 218 214 214
query91 146 148 118 118
query92 142 57 54 54
query93 997 923 583 583
query94 623 396 303 303
query95 382 292 290 290
query96 494 573 284 284
query97 2653 2748 2608 2608
query98 230 213 201 201
query99 1342 1415 1252 1252
Total cold run time: 268006 ms
Total hot run time: 185473 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2834903999
<details>
<summary>TPC-H: <b>Total hot run time: 33990 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 787c34b286f794d7f261bb66d7823155b84c9868, data reload: false
------ Round 1 ----------------------------------
q1 26201 5016 5027 5016
q2 2070 279 183 183
q3 10393 1252 688 688
q4 10218 1027 520 520
q5 7539 2433 2343 2343
q6 179 162 131 131
q7 972 748 596 596
q8 9309 1284 1145 1145
q9 6783 5163 5101 5101
q10 6804 2319 1889 1889
q11 474 285 284 284
q12 351 352 208 208
q13 17771 3673 3098 3098
q14 235 222 212 212
q15 536 478 492 478
q16 430 428 377 377
q17 591 875 372 372
q18 7506 7198 7276 7198
q19 1213 971 560 560
q20 351 334 230 230
q21 4286 2651 2414 2414
q22 1022 1014 947 947
Total cold run time: 115234 ms
Total hot run time: 33990 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5116 5062 5102 5062
q2 235 326 228 228
q3 2167 2676 2244 2244
q4 1398 1781 1375 1375
q5 4528 4447 4411 4411
q6 208 165 128 128
q7 1948 1962 1753 1753
q8 2630 2589 2546 2546
q9 7239 7181 7051 7051
q10 2991 3152 2758 2758
q11 641 505 497 497
q12 670 784 587 587
q13 3553 3843 3267 3267
q14 295 301 263 263
q15 523 497 473 473
q16 431 461 424 424
q17 1131 1535 1410 1410
q18 7777 7485 7419 7419
q19 824 794 878 794
q20 1934 2053 1917 1917
q21 5079 4570 4422 4422
q22 1055 1023 965 965
Total cold run time: 52373 ms
Total hot run time: 49994 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833834934
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823943772
<details>
<summary>ClickBench: <b>Total hot run time: 29.93 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 380ba6cacf44785c79fbf8a3313cb0f6d054a66e, data reload: false
query1 0.04 0.03 0.03
query2 0.13 0.10 0.11
query3 0.26 0.19 0.20
query4 1.60 0.19 0.19
query5 0.59 0.59 0.60
query6 1.19 0.72 0.72
query7 0.03 0.02 0.01
query8 0.04 0.03 0.04
query9 0.56 0.53 0.52
query10 0.58 0.58 0.55
query11 0.16 0.11 0.12
query12 0.14 0.11 0.12
query13 0.60 0.59 0.60
query14 1.15 1.22 1.17
query15 0.87 0.84 0.85
query16 0.39 0.39 0.39
query17 1.02 1.01 1.04
query18 0.21 0.20 0.19
query19 1.88 1.78 1.84
query20 0.02 0.02 0.02
query21 15.40 0.90 0.58
query22 0.74 1.20 0.64
query23 14.99 1.39 0.65
query24 7.52 1.20 1.27
query25 0.48 0.10 0.08
query26 0.67 0.17 0.13
query27 0.06 0.04 0.05
query28 9.79 0.84 0.44
query29 12.53 4.03 3.33
query30 0.26 0.09 0.06
query31 2.81 0.58 0.38
query32 3.23 0.55 0.47
query33 3.08 3.06 3.05
query34 15.73 5.06 4.47
query35 4.54 4.47 4.50
query36 0.67 0.49 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.03
query40 0.17 0.14 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 104.45 s
Total hot run time: 29.93 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2822961903
<details>
<summary>TPC-H: <b>Total hot run time: 33754 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51d565f4fc52c0b7e4d1370e096c79be06110bb0, data reload: false
------ Round 1 ----------------------------------
q1 26177 5000 5027 5000
q2 2059 268 189 189
q3 10396 1209 685 685
q4 10223 996 538 538
q5 7523 2277 2329 2277
q6 177 160 130 130
q7 882 730 603 603
q8 9300 1265 1025 1025
q9 6791 5110 5140 5110
q10 6793 2316 1880 1880
q11 476 277 287 277
q12 351 353 210 210
q13 17750 3650 3050 3050
q14 218 221 215 215
q15 544 479 489 479
q16 448 447 396 396
q17 567 852 347 347
q18 7484 7111 7150 7111
q19 1214 968 545 545
q20 333 351 220 220
q21 3775 3350 2471 2471
q22 1156 1091 996 996
Total cold run time: 114637 ms
Total hot run time: 33754 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5108 5062 5080 5062
q2 261 327 239 239
q3 2234 2740 2384 2384
q4 1407 1799 1353 1353
q5 4386 4383 4426 4383
q6 213 167 127 127
q7 1946 1924 1754 1754
q8 2576 2514 2500 2500
q9 7211 7141 7029 7029
q10 3004 3184 2757 2757
q11 570 508 495 495
q12 673 776 605 605
q13 3484 3894 3239 3239
q14 270 292 273 273
q15 513 493 487 487
q16 478 522 468 468
q17 1122 1532 1394 1394
q18 7665 7557 7494 7494
q19 790 776 836 776
q20 1991 2074 1872 1872
q21 5174 4746 4796 4746
q22 1102 1074 1021 1021
Total cold run time: 52178 ms
Total hot run time: 50458 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2879226202
<details>
<summary>TPC-DS: <b>Total hot run time: 193140 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c17efe382987bbb5fe265707273ae1c879a3a35, data reload: false
query1 1446 1082 1098 1082
query2 6062 1911 1868 1868
query3 10981 4309 4439 4309
query4 55001 25446 23293 23293
query5 4912 614 458 458
query6 327 199 194 194
query7 4873 508 311 311
query8 344 264 261 261
query9 5517 2656 2653 2653
query10 415 336 277 277
query11 15038 14979 14926 14926
query12 166 114 109 109
query13 1039 533 405 405
query14 10176 6493 6338 6338
query15 212 196 188 188
query16 7156 645 461 461
query17 1089 767 595 595
query18 1577 416 317 317
query19 209 193 175 175
query20 129 140 120 120
query21 214 140 110 110
query22 4360 4486 4328 4328
query23 34451 33662 33520 33520
query24 6526 2431 2445 2431
query25 467 485 423 423
query26 719 270 152 152
query27 2355 525 348 348
query28 2951 2151 2140 2140
query29 604 564 440 440
query30 268 220 196 196
query31 818 849 745 745
query32 77 73 72 72
query33 447 375 331 331
query34 764 867 538 538
query35 800 851 759 759
query36 984 996 907 907
query37 115 100 81 81
query38 4263 4278 4204 4204
query39 1547 1487 1449 1449
query40 223 119 113 113
query41 54 53 52 52
query42 124 113 107 107
query43 510 515 511 511
query44 1319 848 839 839
query45 187 186 191 186
query46 845 1035 648 648
query47 1899 1869 1853 1853
query48 395 431 322 322
query49 688 520 456 456
query50 655 721 422 422
query51 4365 4224 4235 4224
query52 112 109 105 105
query53 234 259 193 193
query54 588 591 538 538
query55 87 87 86 86
query56 300 302 294 294
query57 1159 1194 1142 1142
query58 273 283 272 272
query59 2733 2919 2692 2692
query60 341 339 322 322
query61 140 138 160 138
query62 736 732 681 681
query63 227 192 186 186
query64 1908 1044 705 705
query65 4393 4220 4228 4220
query66 718 399 298 298
query67 16126 15845 15357 15357
query68 6760 881 518 518
query69 535 316 269 269
query70 1260 1140 1108 1108
query71 533 325 309 309
query72 5854 4942 5075 4942
query73 1576 735 364 364
query74 8986 9045 8962 8962
query75 4110 3240 2731 2731
query76 4166 1207 765 765
query77 748 361 283 283
query78 9453 9407 8728 8728
query79 2268 814 612 612
query80 567 626 449 449
query81 478 260 218 218
query82 273 128 98 98
query83 251 248 242 242
query84 293 115 83 83
query85 745 348 291 291
query86 366 292 298 292
query87 4549 4538 4326 4326
query88 3749 2273 2294 2273
query89 410 308 292 292
query90 1802 216 208 208
query91 159 148 166 148
query92 78 64 57 57
query93 1829 975 580 580
query94 647 386 316 316
query95 380 324 298 298
query96 499 571 279 279
query97 3168 3234 3143 3143
query98 244 224 210 210
query99 1358 1452 1270 1270
Total cold run time: 298832 ms
Total hot run time: 193140 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890031805
<details>
<summary>TPC-H: <b>Total hot run time: 33853 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 571206ee6a2859721f242331e63007ddc92b9688, data reload: false
------ Round 1 ----------------------------------
q1 26309 5088 5044 5044
q2 2091 288 194 194
q3 10534 1250 706 706
q4 10279 1031 531 531
q5 9093 2386 2386 2386
q6 269 162 131 131
q7 917 748 625 625
q8 9326 1303 1121 1121
q9 6895 5054 5087 5054
q10 6863 2328 1880 1880
q11 483 293 266 266
q12 355 370 216 216
q13 17768 3651 3065 3065
q14 226 231 223 223
q15 534 508 490 490
q16 423 434 378 378
q17 604 875 368 368
q18 7627 7138 7058 7058
q19 1556 970 563 563
q20 326 323 222 222
q21 3806 3214 2378 2378
q22 1059 1027 954 954
Total cold run time: 117343 ms
Total hot run time: 33853 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5158 5055 5089 5055
q2 250 327 227 227
q3 2163 2667 2302 2302
q4 1343 1774 1475 1475
q5 4595 4458 4319 4319
q6 208 164 125 125
q7 1963 1924 1764 1764
q8 2674 2627 2548 2548
q9 7161 7169 7103 7103
q10 2986 3220 2752 2752
q11 580 518 510 510
q12 739 791 646 646
q13 3508 3894 3211 3211
q14 272 309 279 279
q15 526 473 472 472
q16 460 503 444 444
q17 1170 1509 1391 1391
q18 7650 7498 7367 7367
q19 854 880 931 880
q20 1991 1977 1879 1879
q21 4724 4261 4263 4261
q22 1077 1007 1004 1004
Total cold run time: 52052 ms
Total hot run time: 50014 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2872344225
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795763846
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2919087653
<details>
<summary>TPC-H: <b>Total hot run time: 34191 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72063f534c073770ef22ff79f1cb2eb3c505b7e7, data reload: false
------ Round 1 ----------------------------------
q1 26550 5137 5263 5137
q2 1956 285 192 192
q3 10398 1242 712 712
q4 10242 1027 542 542
q5 7682 2456 2334 2334
q6 188 168 140 140
q7 906 738 625 625
q8 9297 1355 1078 1078
q9 6776 5072 5089 5072
q10 6865 2311 1919 1919
q11 512 287 268 268
q12 341 350 218 218
q13 17776 3660 3159 3159
q14 244 238 216 216
q15 562 485 489 485
q16 434 455 376 376
q17 587 858 360 360
q18 7970 7239 7243 7239
q19 1432 955 546 546
q20 334 330 217 217
q21 3926 3250 2377 2377
q22 1041 1013 979 979
Total cold run time: 116019 ms
Total hot run time: 34191 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5138 5184 5055 5055
q2 244 322 222 222
q3 2187 2676 2349 2349
q4 1393 1818 1350 1350
q5 4250 4104 4166 4104
q6 211 164 123 123
q7 1887 1795 1660 1660
q8 2481 2540 2446 2446
q9 6795 6745 6762 6745
q10 2902 3060 2671 2671
q11 574 488 482 482
q12 647 741 578 578
q13 3352 3685 3131 3131
q14 267 302 255 255
q15 522 472 469 469
q16 427 468 433 433
q17 1080 1477 1319 1319
q18 7323 7205 7228 7205
q19 772 789 978 789
q20 1917 2042 1910 1910
q21 4810 4328 4298 4298
q22 1075 1058 995 995
Total cold run time: 50254 ms
Total hot run time: 48589 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3035316907
<details>
<summary>TPC-DS: <b>Total hot run time: 184550 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb22f5a51e780f7c6e69fd2928b66574465f5027, data reload: false
query1 1019 399 409 399
query2 6533 1806 1790 1790
query3 6761 228 216 216
query4 26543 23691 23056 23056
query5 4373 575 448 448
query6 303 214 196 196
query7 4623 491 289 289
query8 280 231 221 221
query9 8638 2669 2701 2669
query10 484 330 284 284
query11 15760 15150 14822 14822
query12 160 106 107 106
query13 1652 552 441 441
query14 9436 5672 5738 5672
query15 244 204 172 172
query16 7575 605 471 471
query17 1173 732 564 564
query18 2009 407 303 303
query19 189 189 156 156
query20 116 112 113 112
query21 216 118 103 103
query22 4102 4045 3895 3895
query23 33987 33024 33185 33024
query24 8467 2402 2382 2382
query25 532 454 396 396
query26 762 273 149 149
query27 2737 503 348 348
query28 4289 2145 2121 2121
query29 679 553 445 445
query30 279 212 201 201
query31 891 809 729 729
query32 75 66 64 64
query33 549 361 304 304
query34 810 852 521 521
query35 770 810 742 742
query36 968 993 900 900
query37 115 94 75 75
query38 4251 4232 4132 4132
query39 1481 1416 1414 1414
query40 209 115 105 105
query41 64 53 52 52
query42 124 107 112 107
query43 517 514 498 498
query44 1384 832 831 831
query45 175 175 167 167
query46 841 1027 637 637
query47 1740 1787 1708 1708
query48 388 417 338 338
query49 715 500 452 452
query50 664 692 417 417
query51 4082 4218 4178 4178
query52 122 105 97 97
query53 265 256 189 189
query54 581 574 511 511
query55 87 77 88 77
query56 314 303 325 303
query57 1204 1183 1111 1111
query58 274 257 269 257
query59 2838 2802 2643 2643
query60 349 307 319 307
query61 128 123 126 123
query62 787 710 656 656
query63 226 195 195 195
query64 3340 1018 664 664
query65 4306 4229 4214 4214
query66 914 420 306 306
query67 15803 15705 15457 15457
query68 8164 925 538 538
query69 481 311 277 277
query70 1235 1128 1125 1125
query71 503 335 293 293
query72 5508 4804 4689 4689
query73 712 603 351 351
query74 8944 9128 8666 8666
query75 3831 3188 2674 2674
query76 3594 1144 724 724
query77 796 385 348 348
query78 9395 9742 8702 8702
query79 2146 858 591 591
query80 582 501 446 446
query81 496 252 224 224
query82 212 125 96 96
query83 249 251 243 243
query84 255 97 88 88
query85 758 362 328 328
query86 380 291 288 288
query87 4440 4512 4377 4377
query88 3788 2303 2265 2265
query89 400 313 284 284
query90 1945 209 198 198
query91 140 146 112 112
query92 72 63 53 53
query93 1566 970 590 590
query94 621 425 305 305
query95 384 290 287 287
query96 504 563 280 280
query97 2655 2779 2704 2704
query98 234 209 206 206
query99 1324 1379 1266 1266
Total cold run time: 272586 ms
Total hot run time: 184550 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3052453033
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2804556187
<details>
<summary>ClickBench: <b>Total hot run time: 30.82 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2930d988d2c992f3fe1a6254e06d831d20a0a171, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.11
query3 0.26 0.19 0.19
query4 1.60 0.20 0.21
query5 0.59 0.57 0.58
query6 1.18 0.72 0.72
query7 0.02 0.02 0.01
query8 0.04 0.04 0.03
query9 0.57 0.52 0.51
query10 0.56 0.56 0.57
query11 0.15 0.11 0.11
query12 0.14 0.11 0.11
query13 0.62 0.59 0.60
query14 2.70 2.82 2.79
query15 0.94 0.88 0.86
query16 0.38 0.37 0.38
query17 1.02 1.02 1.00
query18 0.21 0.20 0.19
query19 1.88 1.96 1.78
query20 0.01 0.01 0.01
query21 15.35 0.97 0.59
query22 0.75 1.18 0.64
query23 15.02 1.40 0.66
query24 7.00 1.76 0.46
query25 0.45 0.07 0.21
query26 0.60 0.16 0.13
query27 0.05 0.05 0.05
query28 10.19 0.94 0.43
query29 12.58 3.96 3.27
query30 0.25 0.09 0.07
query31 2.81 0.58 0.39
query32 3.22 0.55 0.47
query33 3.11 3.00 3.00
query34 15.79 5.15 4.52
query35 4.48 4.52 4.55
query36 0.66 0.49 0.48
query37 0.09 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 105.87 s
Total hot run time: 30.82 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2809250109
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3117183419
<details>
<summary>TPC-H: <b>Total hot run time: 34039 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3f0b61e7e8cc8de1b85cb8c4ac561353c78b15d, data reload: false
------ Round 1 ----------------------------------
q1 17606 5268 5169 5169
q2 1992 279 177 177
q3 10500 1260 716 716
q4 10246 1009 523 523
q5 8055 2402 2373 2373
q6 182 160 125 125
q7 883 748 595 595
q8 9312 1321 1116 1116
q9 7078 5131 5134 5131
q10 6876 2361 1955 1955
q11 491 293 283 283
q12 344 355 215 215
q13 17765 3707 3077 3077
q14 217 229 214 214
q15 556 488 503 488
q16 423 422 369 369
q17 590 880 357 357
q18 7459 7037 7210 7037
q19 1263 967 545 545
q20 340 340 227 227
q21 3988 3187 2369 2369
q22 1072 1043 978 978
Total cold run time: 107238 ms
Total hot run time: 34039 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5204 5118 5182 5118
q2 247 309 217 217
q3 2209 2682 2270 2270
q4 1324 1805 1332 1332
q5 4424 4453 4511 4453
q6 213 167 130 130
q7 2028 1970 1736 1736
q8 2700 2642 2594 2594
q9 7336 7391 7379 7379
q10 3108 3284 2919 2919
q11 585 510 509 509
q12 690 798 642 642
q13 3612 4187 3429 3429
q14 282 326 269 269
q15 520 472 478 472
q16 446 480 456 456
q17 1162 1545 1448 1448
q18 7874 8055 7509 7509
q19 830 843 914 843
q20 1961 1973 1794 1794
q21 4823 4415 4247 4247
q22 1074 1034 1005 1005
Total cold run time: 52652 ms
Total hot run time: 50771 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2944380580
# FE UT Coverage Report
Increment line coverage ` 83.25% (318/382)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_c0c5557ef07d1fbe8f42072b38acc17bfaed12bd/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_c0c5557ef07d1fbe8f42072b38acc17bfaed12bd/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3086639634
<details>
<summary>TPC-DS: <b>Total hot run time: 190329 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb92819243a981911b585dcecfc6e8655c8959c7, data reload: false
query1 1001 395 422 395
query2 6554 1746 1736 1736
query3 6757 235 230 230
query4 27032 23711 23473 23473
query5 4389 652 511 511
query6 306 244 219 219
query7 4651 525 311 311
query8 389 340 328 328
query9 8689 2680 2694 2680
query10 498 374 310 310
query11 15788 15103 14816 14816
query12 192 131 124 124
query13 1725 586 448 448
query14 10145 6100 6073 6073
query15 242 220 175 175
query16 7655 653 467 467
query17 1358 724 597 597
query18 2047 449 341 341
query19 201 224 173 173
query20 141 133 124 124
query21 241 147 124 124
query22 4431 4659 4589 4589
query23 35002 34283 33433 33433
query24 8402 2391 2392 2391
query25 531 451 403 403
query26 1230 271 156 156
query27 2716 524 350 350
query28 4322 2192 2162 2162
query29 757 567 449 449
query30 296 229 215 215
query31 929 838 760 760
query32 84 74 76 74
query33 611 378 337 337
query34 822 848 556 556
query35 806 863 770 770
query36 998 1028 923 923
query37 127 110 83 83
query38 4245 4118 4105 4105
query39 1498 1457 1659 1457
query40 231 134 127 127
query41 107 101 98 98
query42 135 113 116 113
query43 518 532 486 486
query44 1339 837 855 837
query45 185 185 174 174
query46 868 1023 647 647
query47 1794 1833 1765 1765
query48 414 446 328 328
query49 815 587 469 469
query50 679 709 438 438
query51 5576 5607 5502 5502
query52 121 116 106 106
query53 248 278 210 210
query54 607 609 531 531
query55 91 85 90 85
query56 348 335 310 310
query57 1196 1233 1141 1141
query58 308 301 318 301
query59 2526 2652 2530 2530
query60 357 390 327 327
query61 157 146 151 146
query62 826 756 676 676
query63 244 206 207 206
query64 4308 1066 725 725
query65 4284 4202 4187 4187
query66 1259 596 483 483
query67 16007 15828 15623 15623
query68 8242 935 559 559
query69 494 354 307 307
query70 1247 1163 1155 1155
query71 509 360 324 324
query72 5658 4750 4795 4750
query73 726 631 384 384
query74 9240 9331 8688 8688
query75 3911 3258 2823 2823
query76 3717 1161 729 729
query77 857 462 366 366
query78 9577 9872 9000 9000
query79 2119 864 608 608
query80 724 581 513 513
query81 483 289 241 241
query82 237 141 108 108
query83 317 294 272 272
query84 372 109 96 96
query85 801 409 367 367
query86 362 332 308 308
query87 4500 4465 4416 4416
query88 3084 2438 2400 2400
query89 422 338 314 314
query90 2114 233 239 233
query91 156 160 134 134
query92 92 76 71 71
query93 1346 974 604 604
query94 693 430 336 336
query95 400 309 325 309
query96 520 582 293 293
query97 2698 2807 2687 2687
query98 255 223 213 213
query99 1443 1447 1282 1282
Total cold run time: 280672 ms
Total hot run time: 190329 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3086650757
<details>
<summary>ClickBench: <b>Total hot run time: 32.91 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb92819243a981911b585dcecfc6e8655c8959c7, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.05 0.04
query3 0.24 0.08 0.08
query4 1.61 0.11 0.10
query5 0.42 0.42 0.41
query6 1.17 0.67 0.67
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.59 0.53 0.50
query10 0.58 0.57 0.59
query11 0.16 0.11 0.12
query12 0.15 0.12 0.13
query13 0.63 0.63 0.62
query14 0.80 0.83 0.82
query15 0.90 0.87 0.87
query16 0.38 0.38 0.41
query17 1.05 1.05 1.05
query18 0.23 0.22 0.21
query19 1.93 1.84 1.85
query20 0.02 0.02 0.01
query21 15.40 0.93 0.54
query22 0.76 1.37 0.75
query23 14.76 1.38 0.64
query24 7.67 0.87 0.94
query25 0.53 0.16 0.07
query26 0.55 0.17 0.14
query27 0.07 0.06 0.06
query28 9.38 0.92 0.46
query29 12.57 3.92 3.35
query30 3.16 3.09 3.02
query31 2.83 0.59 0.39
query32 3.24 0.56 0.48
query33 3.13 3.12 3.17
query34 16.10 5.37 4.78
query35 4.83 4.84 4.83
query36 0.69 0.51 0.52
query37 0.09 0.08 0.07
query38 0.06 0.05 0.05
query39 0.04 0.03 0.03
query40 0.19 0.15 0.16
query41 0.08 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 107.26 s
Total hot run time: 32.91 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209105822
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
Review Comment:
had fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076541140
<details>
<summary>TPC-DS: <b>Total hot run time: 187109 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 96e01840ac51fd74427a6987028ab866c386ed9f, data reload: false
query1 1007 380 381 380
query2 6536 1753 1669 1669
query3 6746 212 206 206
query4 26919 23690 23530 23530
query5 4320 568 425 425
query6 284 208 208 208
query7 4619 498 287 287
query8 271 226 222 222
query9 8663 2671 2669 2669
query10 479 349 268 268
query11 15232 15122 14916 14916
query12 159 104 101 101
query13 1634 532 396 396
query14 8695 5968 5904 5904
query15 205 196 170 170
query16 7118 613 456 456
query17 910 674 567 567
query18 1962 408 299 299
query19 192 179 162 162
query20 118 120 114 114
query21 214 128 105 105
query22 4169 4402 4059 4059
query23 34092 33312 33217 33217
query24 8481 2407 2391 2391
query25 581 496 433 433
query26 1248 270 156 156
query27 2768 509 356 356
query28 4387 2143 2124 2124
query29 783 582 485 485
query30 293 223 195 195
query31 885 809 683 683
query32 77 69 65 65
query33 587 361 337 337
query34 817 871 510 510
query35 764 826 755 755
query36 1003 985 911 911
query37 109 103 79 79
query38 4273 4236 4080 4080
query39 1502 1448 1441 1441
query40 210 117 106 106
query41 62 57 53 53
query42 122 104 106 104
query43 519 513 493 493
query44 1294 822 822 822
query45 174 178 166 166
query46 835 1019 619 619
query47 1777 1840 1764 1764
query48 378 408 300 300
query49 746 461 399 399
query50 655 674 444 444
query51 5514 5623 5400 5400
query52 106 108 102 102
query53 226 261 179 179
query54 566 564 486 486
query55 85 79 82 79
query56 314 318 284 284
query57 1197 1224 1161 1161
query58 271 260 261 260
query59 2534 2705 2537 2537
query60 326 325 312 312
query61 133 128 127 127
query62 788 727 650 650
query63 226 193 190 190
query64 4350 1003 666 666
query65 4249 4186 4170 4170
query66 1134 426 313 313
query67 15800 15845 15647 15647
query68 7972 890 530 530
query69 474 313 273 273
query70 1218 1181 1158 1158
query71 475 315 320 315
query72 5743 4835 4765 4765
query73 718 629 350 350
query74 9111 9116 8810 8810
query75 3812 3204 2657 2657
query76 3671 1159 738 738
query77 783 376 299 299
query78 9428 9372 8718 8718
query79 2400 828 580 580
query80 588 509 443 443
query81 505 263 229 229
query82 469 126 97 97
query83 258 250 241 241
query84 291 123 90 90
query85 809 403 358 358
query86 399 319 320 319
query87 4570 4465 4417 4417
query88 3485 2311 2310 2310
query89 387 327 290 290
query90 1880 215 216 215
query91 145 146 113 113
query92 81 64 58 58
query93 1744 989 590 590
query94 680 416 305 305
query95 377 293 288 288
query96 499 573 281 281
query97 2676 2816 2609 2609
query98 243 216 203 203
query99 1350 1434 1296 1296
Total cold run time: 274934 ms
Total hot run time: 187109 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209095699
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
Review Comment:
at earlier, put constant propagation at the begin of rewrite phase, and now put it later than apply
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3082355080
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2978835849
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2978835212
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2872509276
<details>
<summary>ClickBench: <b>Total hot run time: 29.63 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ea6e2dde86b7743e3f5550e1f93c4874995b9b05, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.12
query3 0.26 0.19 0.20
query4 1.59 0.20 0.11
query5 0.56 0.56 0.55
query6 1.21 0.73 0.72
query7 0.03 0.01 0.02
query8 0.04 0.04 0.03
query9 0.56 0.55 0.50
query10 0.58 0.56 0.58
query11 0.16 0.12 0.11
query12 0.16 0.11 0.12
query13 0.61 0.60 0.59
query14 0.79 0.81 0.79
query15 0.87 0.87 0.85
query16 0.37 0.39 0.40
query17 1.05 1.10 1.04
query18 0.23 0.21 0.21
query19 1.87 1.84 1.85
query20 0.02 0.01 0.01
query21 15.41 0.88 0.54
query22 0.74 1.32 0.75
query23 14.78 1.36 0.62
query24 6.93 1.12 1.39
query25 0.50 0.12 0.14
query26 0.53 0.16 0.13
query27 0.05 0.05 0.06
query28 10.53 0.88 0.44
query29 12.56 3.94 3.29
query30 0.25 0.09 0.06
query31 2.84 0.60 0.38
query32 3.23 0.57 0.47
query33 3.04 3.11 3.09
query34 15.87 5.10 4.49
query35 4.49 4.53 4.51
query36 0.66 0.50 0.49
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.18 0.15 0.14
query41 0.08 0.04 0.03
query42 0.03 0.02 0.02
query43 0.04 0.04 0.02
Total cold run time: 104.05 s
Total hot run time: 29.63 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2875961648
<details>
<summary>TPC-DS: <b>Total hot run time: 192051 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d39a11d88dea7be0865d4d4061a50ab3d8b34a69, data reload: false
query1 1398 1078 1079 1078
query2 6291 1826 1766 1766
query3 11001 4474 4263 4263
query4 53350 24759 23329 23329
query5 5228 520 468 468
query6 355 211 202 202
query7 5039 507 296 296
query8 323 255 248 248
query9 6056 2653 2659 2653
query10 446 321 268 268
query11 15069 15378 14816 14816
query12 156 112 107 107
query13 1118 536 411 411
query14 10107 6184 6275 6184
query15 195 228 185 185
query16 6990 632 485 485
query17 1054 733 569 569
query18 1521 419 327 327
query19 203 202 175 175
query20 137 135 123 123
query21 210 132 124 124
query22 4564 4554 4515 4515
query23 34265 33398 33744 33398
query24 6988 2404 2446 2404
query25 507 485 421 421
query26 673 281 149 149
query27 2190 514 334 334
query28 3428 2142 2141 2141
query29 609 574 437 437
query30 278 222 204 204
query31 845 812 747 747
query32 75 64 62 62
query33 465 363 331 331
query34 784 878 531 531
query35 770 889 755 755
query36 939 1007 886 886
query37 121 101 79 79
query38 4204 4363 4111 4111
query39 1482 1455 1451 1451
query40 212 115 104 104
query41 54 59 52 52
query42 152 118 121 118
query43 484 492 479 479
query44 1323 844 831 831
query45 186 175 169 169
query46 846 1020 647 647
query47 1914 1876 1845 1845
query48 397 439 329 329
query49 712 508 440 440
query50 654 699 409 409
query51 4229 4188 4200 4188
query52 109 111 101 101
query53 218 256 185 185
query54 635 576 535 535
query55 87 84 89 84
query56 319 321 301 301
query57 1195 1212 1113 1113
query58 268 269 260 260
query59 2645 2780 2601 2601
query60 369 336 311 311
query61 128 125 134 125
query62 744 734 689 689
query63 230 187 198 187
query64 1445 1057 677 677
query65 4311 4222 4197 4197
query66 722 397 300 300
query67 16194 15804 15527 15527
query68 5466 903 530 530
query69 522 302 276 276
query70 1188 1141 1140 1140
query71 446 315 315 315
query72 5854 4892 4747 4747
query73 1198 662 373 373
query74 9282 9328 8598 8598
query75 3383 3255 2719 2719
query76 3774 1180 747 747
query77 570 356 293 293
query78 9386 9626 8728 8728
query79 999 886 595 595
query80 653 514 442 442
query81 480 247 233 233
query82 283 126 95 95
query83 258 255 255 255
query84 294 105 86 86
query85 715 325 290 290
query86 338 308 303 303
query87 4511 4422 4431 4422
query88 2969 2316 2341 2316
query89 391 328 288 288
query90 1687 214 215 214
query91 148 143 118 118
query92 66 61 55 55
query93 1122 963 586 586
query94 621 414 297 297
query95 381 294 281 281
query96 501 587 284 284
query97 3134 3257 3094 3094
query98 220 220 217 217
query99 1346 1413 1277 1277
Total cold run time: 292402 ms
Total hot run time: 192051 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2976194690
<details>
<summary>TPC-DS: <b>Total hot run time: 194747 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92f51c7b47bfdc21b0d37aaab60c326abc3681d8, data reload: false
query1 1432 1016 996 996
query2 6303 1931 1906 1906
query3 11018 4480 4641 4480
query4 54467 26435 23424 23424
query5 5184 548 491 491
query6 361 236 216 216
query7 4966 545 343 343
query8 312 253 226 226
query9 6005 2855 2889 2855
query10 473 349 295 295
query11 15144 15190 14948 14948
query12 161 116 112 112
query13 1101 562 437 437
query14 10225 6436 6459 6436
query15 199 210 189 189
query16 7171 629 508 508
query17 1092 727 575 575
query18 1595 419 314 314
query19 212 215 185 185
query20 137 135 126 126
query21 210 130 114 114
query22 4408 4448 4309 4309
query23 34424 33611 33552 33552
query24 6825 2472 2486 2472
query25 512 482 450 450
query26 704 282 159 159
query27 2262 557 369 369
query28 3051 2359 2331 2331
query29 593 589 458 458
query30 273 229 196 196
query31 864 889 735 735
query32 75 73 74 73
query33 467 403 327 327
query34 813 899 554 554
query35 814 903 792 792
query36 1024 1057 943 943
query37 121 104 82 82
query38 4208 4343 4213 4213
query39 1552 1487 1470 1470
query40 212 126 118 118
query41 63 68 63 63
query42 132 128 121 121
query43 536 535 526 526
query44 1477 903 906 903
query45 196 182 173 173
query46 908 1088 683 683
query47 1905 1912 1799 1799
query48 417 471 345 345
query49 670 512 423 423
query50 727 731 437 437
query51 4190 4244 4262 4244
query52 124 120 112 112
query53 256 271 210 210
query54 634 618 563 563
query55 96 96 97 96
query56 327 344 342 342
query57 1230 1240 1172 1172
query58 363 277 270 270
query59 2731 2843 2701 2701
query60 358 336 342 336
query61 131 129 130 129
query62 738 742 679 679
query63 239 198 202 198
query64 1652 1130 714 714
query65 4274 4196 4124 4124
query66 711 410 314 314
query67 16045 15725 15580 15580
query68 7346 934 574 574
query69 554 347 281 281
query70 1248 1146 1147 1146
query71 495 353 322 322
query72 5828 4764 4738 4738
query73 1204 671 384 384
query74 9445 9190 9011 9011
query75 3792 3241 2706 2706
query76 4096 1204 772 772
query77 653 387 296 296
query78 9474 9343 8708 8708
query79 1975 871 624 624
query80 624 556 488 488
query81 473 261 235 235
query82 444 132 107 107
query83 275 267 245 245
query84 294 124 97 97
query85 905 378 327 327
query86 392 340 311 311
query87 4355 4447 4447 4447
query88 3701 2471 2449 2449
query89 404 333 297 297
query90 1890 217 210 210
query91 148 151 122 122
query92 88 63 57 57
query93 1365 1002 631 631
query94 685 419 307 307
query95 379 309 314 309
query96 543 599 296 296
query97 2762 2744 2716 2716
query98 259 216 216 216
query99 1689 1433 1282 1282
Total cold run time: 299885 ms
Total hot run time: 194747 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "924060929 (via GitHub)" <gi...@apache.org>.
924060929 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2194851144
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
Review Comment:
change to `plan.containsType(LogicalApply.class)` for speed
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ColumnPruning.java:
##########
@@ -227,6 +227,13 @@ public Plan visitLogicalUnion(LogicalUnion union, PruneContext context) {
.filter(index -> prunedOutput.contains(originOutput.get(index)))
.boxed()
.collect(ImmutableList.toImmutableList());
+ // in function `pruneUnionOutput`, the prunedOutput may not come from originOutput,
+ // it's just a new add column slot reference.
+ if (prunedOutputIndexes.isEmpty()) {
+ prunedOutputIndexes = IntStream.range(0, prunedOutput.size())
+ .boxed()
+ .collect(ImmutableList.toImmutableList());
+ }
Review Comment:
provide the detail of the case in the comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890065904
<details>
<summary>TPC-DS: <b>Total hot run time: 185061 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 571206ee6a2859721f242331e63007ddc92b9688, data reload: false
query1 1025 487 491 487
query2 6570 1914 1893 1893
query3 6744 248 220 220
query4 26166 23612 22884 22884
query5 4359 629 462 462
query6 315 234 208 208
query7 4624 488 298 298
query8 319 275 256 256
query9 8631 2609 2625 2609
query10 494 330 270 270
query11 15314 15154 14736 14736
query12 156 107 103 103
query13 1664 521 407 407
query14 9222 5988 6162 5988
query15 222 190 179 179
query16 7192 624 427 427
query17 1220 719 579 579
query18 1981 414 321 321
query19 201 194 171 171
query20 120 117 119 117
query21 218 150 115 115
query22 4298 4377 4250 4250
query23 34105 33143 33009 33009
query24 8474 2395 2399 2395
query25 590 490 431 431
query26 1238 272 155 155
query27 2738 502 350 350
query28 4329 2133 2109 2109
query29 752 553 442 442
query30 287 222 193 193
query31 922 805 703 703
query32 73 61 67 61
query33 577 379 317 317
query34 813 834 530 530
query35 765 805 746 746
query36 959 981 887 887
query37 111 102 90 90
query38 4069 4089 4108 4089
query39 1510 1619 1419 1419
query40 213 120 109 109
query41 62 57 55 55
query42 123 121 113 113
query43 531 519 491 491
query44 1315 825 836 825
query45 176 179 167 167
query46 852 1047 634 634
query47 1795 1818 1751 1751
query48 397 420 310 310
query49 767 503 427 427
query50 662 680 413 413
query51 4204 4129 4074 4074
query52 111 107 98 98
query53 222 261 180 180
query54 568 575 496 496
query55 86 86 87 86
query56 325 303 307 303
query57 1143 1156 1098 1098
query58 272 273 273 273
query59 2722 2786 2698 2698
query60 332 337 317 317
query61 128 142 163 142
query62 774 720 672 672
query63 223 189 190 189
query64 4364 1007 671 671
query65 4357 4214 4254 4214
query66 1141 414 310 310
query67 15806 15788 15484 15484
query68 8761 866 523 523
query69 467 322 279 279
query70 1214 1103 1120 1103
query71 477 326 318 318
query72 5334 4542 4715 4542
query73 712 610 360 360
query74 8833 8991 8733 8733
query75 4239 3182 2714 2714
query76 3676 1189 768 768
query77 797 385 288 288
query78 9486 9466 8714 8714
query79 1880 878 587 587
query80 584 505 513 505
query81 467 260 220 220
query82 420 130 94 94
query83 283 253 235 235
query84 297 103 81 81
query85 767 329 285 285
query86 338 306 280 280
query87 4410 4408 4343 4343
query88 3099 2344 2343 2343
query89 389 316 280 280
query90 1939 211 209 209
query91 145 150 116 116
query92 70 62 59 59
query93 1039 937 577 577
query94 670 397 301 301
query95 367 294 285 285
query96 504 576 289 289
query97 2716 2781 2636 2636
query98 233 207 209 207
query99 1445 1424 1281 1281
Total cold run time: 273096 ms
Total hot run time: 185061 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2826879403
<details>
<summary>TPC-H: <b>Total hot run time: 33914 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 567137845144acf42624ab070da50afc03607d57, data reload: false
------ Round 1 ----------------------------------
q1 26198 5052 5004 5004
q2 2065 287 180 180
q3 10386 1265 690 690
q4 10215 1018 552 552
q5 7598 2399 2350 2350
q6 178 165 131 131
q7 939 733 641 641
q8 9341 1301 1118 1118
q9 6769 5029 5028 5028
q10 6880 2313 1897 1897
q11 460 284 275 275
q12 345 357 221 221
q13 17766 3701 3116 3116
q14 233 221 217 217
q15 523 484 482 482
q16 438 441 395 395
q17 610 860 363 363
q18 7511 7101 7069 7069
q19 1423 960 563 563
q20 342 331 220 220
q21 4094 2690 2452 2452
q22 1032 1002 950 950
Total cold run time: 115346 ms
Total hot run time: 33914 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5086 5074 5054 5054
q2 242 330 228 228
q3 2169 2657 2277 2277
q4 1429 1803 1407 1407
q5 4525 4504 4500 4500
q6 216 169 129 129
q7 1982 1924 1738 1738
q8 2562 2647 2606 2606
q9 7220 7132 7330 7132
q10 3023 3194 2745 2745
q11 566 503 479 479
q12 657 771 629 629
q13 3462 3845 3249 3249
q14 275 283 275 275
q15 523 480 484 480
q16 461 491 459 459
q17 1154 1588 1379 1379
q18 7729 7440 7399 7399
q19 840 777 889 777
q20 1989 2002 1868 1868
q21 5277 4768 4595 4595
q22 1051 999 953 953
Total cold run time: 52438 ms
Total hot run time: 50358 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3073919679
<details>
<summary>TPC-H: <b>Total hot run time: 33113 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8155bdac55ae0d1587c99381e8f178452066e663, data reload: false
------ Round 1 ----------------------------------
q1 17563 5293 5024 5024
q2 1925 284 195 195
q3 10286 1253 712 712
q4 10228 1008 529 529
q5 7508 2372 2266 2266
q6 184 159 132 132
q7 876 733 612 612
q8 9305 1263 1088 1088
q9 6804 5125 5124 5124
q10 6878 2385 1973 1973
q11 479 290 264 264
q12 334 350 211 211
q13 17756 3687 3154 3154
q14 222 229 213 213
q15 539 490 478 478
q16 422 434 370 370
q17 564 859 347 347
q18 7389 7147 7003 7003
q19 1211 942 558 558
q20 354 341 213 213
q21 3863 3176 2346 2346
q22 360 316 301 301
Total cold run time: 105050 ms
Total hot run time: 33113 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5143 5150 5107 5107
q2 241 334 229 229
q3 2148 2612 2280 2280
q4 1336 1725 1335 1335
q5 4157 4104 4506 4104
q6 215 168 124 124
q7 2014 1984 1812 1812
q8 2660 2738 2599 2599
q9 7205 7201 7249 7201
q10 3179 3287 2931 2931
q11 591 494 489 489
q12 688 816 680 680
q13 3699 3970 3393 3393
q14 280 310 281 281
q15 573 482 470 470
q16 443 495 456 456
q17 1176 1546 1464 1464
q18 7997 7781 7406 7406
q19 757 776 881 776
q20 1902 2011 1807 1807
q21 4734 4306 4227 4227
q22 631 563 570 563
Total cold run time: 51769 ms
Total hot run time: 49734 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2914702922
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2914918938
<details>
<summary>ClickBench: <b>Total hot run time: 28.97 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5de5b4e8038ca6b71c5470481e59e65c6de3bb9f, data reload: false
query1 0.03 0.04 0.02
query2 0.12 0.10 0.12
query3 0.24 0.19 0.20
query4 1.59 0.20 0.11
query5 0.43 0.42 0.44
query6 1.18 0.66 0.66
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.59 0.52 0.52
query10 0.59 0.57 0.57
query11 0.16 0.11 0.11
query12 0.14 0.12 0.13
query13 0.61 0.59 0.60
query14 0.79 0.80 0.84
query15 0.87 0.87 0.85
query16 0.39 0.39 0.37
query17 1.04 1.05 1.01
query18 0.23 0.21 0.21
query19 1.98 1.79 1.86
query20 0.01 0.01 0.02
query21 15.39 0.89 0.53
query22 0.75 1.19 0.78
query23 14.75 1.36 0.63
query24 6.82 1.40 0.65
query25 0.47 0.24 0.09
query26 0.54 0.17 0.14
query27 0.05 0.05 0.05
query28 9.17 0.90 0.44
query29 12.56 4.02 3.29
query30 0.25 0.09 0.06
query31 2.82 0.60 0.41
query32 3.22 0.55 0.49
query33 3.03 3.05 3.15
query34 15.76 5.12 4.52
query35 4.57 4.58 4.52
query36 0.66 0.50 0.48
query37 0.09 0.06 0.06
query38 0.05 0.05 0.04
query39 0.03 0.02 0.02
query40 0.18 0.14 0.13
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.05 0.03 0.03
Total cold run time: 102.39 s
Total hot run time: 28.97 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795753837
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795849620
<details>
<summary>ClickBench: <b>Total hot run time: 31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f28a750f710b6d3e443dd61dcdbbcede2b504357, data reload: false
query1 0.04 0.03 0.03
query2 0.13 0.10 0.10
query3 0.26 0.19 0.19
query4 1.59 0.19 0.11
query5 0.56 0.54 0.54
query6 1.20 0.72 0.71
query7 0.02 0.01 0.01
query8 0.04 0.04 0.04
query9 0.58 0.52 0.52
query10 0.59 0.61 0.59
query11 0.15 0.11 0.11
query12 0.14 0.11 0.12
query13 0.62 0.60 0.60
query14 2.67 2.79 2.80
query15 0.93 0.86 0.84
query16 0.40 0.37 0.38
query17 1.05 1.03 1.02
query18 0.22 0.19 0.19
query19 2.05 1.87 1.83
query20 0.01 0.01 0.02
query21 15.36 0.88 0.56
query22 0.75 1.43 0.73
query23 14.76 1.45 0.66
query24 6.81 1.32 0.51
query25 0.50 0.24 0.18
query26 0.71 0.17 0.14
query27 0.05 0.05 0.05
query28 8.61 0.96 0.44
query29 12.53 3.98 3.31
query30 0.25 0.10 0.07
query31 2.81 0.58 0.40
query32 3.23 0.55 0.47
query33 3.00 3.02 3.03
query34 15.70 5.05 4.45
query35 4.55 4.49 4.51
query36 0.65 0.51 0.48
query37 0.09 0.07 0.07
query38 0.05 0.04 0.05
query39 0.03 0.02 0.02
query40 0.17 0.14 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.04 0.03
Total cold run time: 104 s
Total hot run time: 31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2796695592
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2798459717
<details>
<summary>TPC-DS: <b>Total hot run time: 192177 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 679002894d5fb311f725de08d1ffda48e95234fc, data reload: false
query1 1444 1099 1040 1040
query2 6374 1879 1879 1879
query3 10971 4518 4340 4340
query4 53576 25477 23557 23557
query5 5040 517 491 491
query6 341 215 207 207
query7 4877 499 282 282
query8 333 262 260 260
query9 5471 2590 2592 2590
query10 420 334 262 262
query11 14901 14894 14833 14833
query12 158 112 110 110
query13 1040 536 384 384
query14 10120 6436 6314 6314
query15 204 201 186 186
query16 7060 680 465 465
query17 1062 750 617 617
query18 1554 412 312 312
query19 197 197 167 167
query20 133 127 130 127
query21 209 124 107 107
query22 4511 4415 4265 4265
query23 34247 33527 33408 33408
query24 6614 2420 2421 2420
query25 483 473 397 397
query26 680 279 163 163
query27 2288 525 335 335
query28 3115 2465 2431 2431
query29 570 597 457 457
query30 284 227 206 206
query31 817 857 719 719
query32 75 64 63 63
query33 473 375 330 330
query34 776 862 526 526
query35 811 838 751 751
query36 949 1012 887 887
query37 124 100 81 81
query38 4267 4272 4205 4205
query39 1472 1436 1444 1436
query40 214 131 114 114
query41 62 59 59 59
query42 126 113 111 111
query43 514 512 496 496
query44 1341 844 864 844
query45 188 176 169 169
query46 875 1034 646 646
query47 1877 1889 1792 1792
query48 371 413 310 310
query49 696 527 432 432
query50 662 699 410 410
query51 4267 4344 4166 4166
query52 109 103 106 103
query53 239 253 194 194
query54 583 596 540 540
query55 86 82 83 82
query56 311 329 300 300
query57 1199 1210 1143 1143
query58 274 272 268 268
query59 2727 2784 2735 2735
query60 340 333 322 322
query61 135 130 127 127
query62 768 734 663 663
query63 226 193 186 186
query64 1506 1062 685 685
query65 4422 4246 4229 4229
query66 704 393 297 297
query67 15959 15474 15289 15289
query68 4572 894 524 524
query69 509 311 273 273
query70 1168 1116 1071 1071
query71 449 311 296 296
query72 5717 4859 4870 4859
query73 781 605 344 344
query74 8967 9110 9018 9018
query75 3485 3239 2746 2746
query76 3439 1092 650 650
query77 561 466 284 284
query78 9059 9215 8339 8339
query79 1712 884 568 568
query80 662 521 449 449
query81 477 259 237 237
query82 231 129 95 95
query83 254 248 241 241
query84 296 104 85 85
query85 687 330 275 275
query86 319 310 289 289
query87 4406 4475 4313 4313
query88 2782 2222 2298 2222
query89 387 331 279 279
query90 1837 212 213 212
query91 154 140 112 112
query92 75 67 58 58
query93 2198 974 567 567
query94 652 423 304 304
query95 370 295 297 295
query96 490 582 276 276
query97 3155 3236 3119 3119
query98 236 205 205 205
query99 1301 1396 1250 1250
Total cold run time: 290210 ms
Total hot run time: 192177 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833375414
<details>
<summary>TPC-DS: <b>Total hot run time: 185270 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 484c6bfb7960914a92b656959003939aa69b130c, data reload: false
query1 1015 467 483 467
query2 6568 1912 1908 1908
query3 6754 214 211 211
query4 26161 23305 23103 23103
query5 4357 635 446 446
query6 300 199 217 199
query7 4627 482 276 276
query8 291 263 252 252
query9 8663 2565 2598 2565
query10 491 327 278 278
query11 15938 15012 14817 14817
query12 160 106 101 101
query13 1645 501 387 387
query14 9222 6171 6145 6145
query15 221 186 166 166
query16 7195 660 508 508
query17 1208 742 588 588
query18 1971 412 315 315
query19 197 203 164 164
query20 126 123 123 123
query21 213 137 109 109
query22 4044 4274 4134 4134
query23 33842 32841 32981 32841
query24 8541 2371 2377 2371
query25 535 438 387 387
query26 1231 278 155 155
query27 2753 496 329 329
query28 4333 2118 2107 2107
query29 782 589 432 432
query30 286 216 193 193
query31 919 799 734 734
query32 73 65 66 65
query33 560 367 306 306
query34 782 853 521 521
query35 781 815 724 724
query36 952 991 888 888
query37 130 103 77 77
query38 4216 4069 4139 4069
query39 1470 1418 1380 1380
query40 208 114 106 106
query41 59 55 58 55
query42 122 112 109 109
query43 501 503 489 489
query44 1293 807 800 800
query45 181 191 167 167
query46 840 1037 623 623
query47 1764 1794 1742 1742
query48 375 404 303 303
query49 781 518 439 439
query50 677 682 416 416
query51 4165 4199 4070 4070
query52 108 104 96 96
query53 228 252 186 186
query54 562 572 502 502
query55 90 78 79 78
query56 316 296 304 296
query57 1138 1165 1073 1073
query58 264 255 252 252
query59 2674 2836 2637 2637
query60 329 345 310 310
query61 136 128 127 127
query62 806 735 685 685
query63 233 191 191 191
query64 4472 1095 709 709
query65 4478 4230 4258 4230
query66 1148 409 300 300
query67 15678 15411 15325 15325
query68 9344 893 530 530
query69 475 306 263 263
query70 1187 1146 1106 1106
query71 464 331 306 306
query72 5253 4644 4744 4644
query73 717 581 347 347
query74 8812 9120 9015 9015
query75 4238 3211 2705 2705
query76 3886 1258 780 780
query77 1012 381 279 279
query78 9388 9383 8642 8642
query79 2300 876 567 567
query80 604 497 435 435
query81 464 251 225 225
query82 435 124 99 99
query83 279 256 235 235
query84 307 106 90 90
query85 757 331 283 283
query86 340 299 271 271
query87 4366 4510 4292 4292
query88 2877 2217 2244 2217
query89 384 317 300 300
query90 1931 210 215 210
query91 138 140 111 111
query92 72 62 56 56
query93 1074 959 586 586
query94 660 419 305 305
query95 380 291 295 291
query96 488 562 273 273
query97 3187 3260 3151 3151
query98 230 210 210 210
query99 1433 1423 1289 1289
Total cold run time: 274661 ms
Total hot run time: 185270 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3082484723
# FE UT Coverage Report
Increment line coverage ` 83.21% (342/411)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_f0980ee784b099b4efc03bfc83b22506ce899b13/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_f0980ee784b099b4efc03bfc83b22506ce899b13/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3077870907
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3077815530
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3079286871
<details>
<summary>TPC-DS: <b>Total hot run time: 187418 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a20e730c75a1273b1167641933d055aea6470f95, data reload: false
query1 992 412 412 412
query2 6557 1767 1707 1707
query3 6739 210 209 209
query4 26773 23743 23256 23256
query5 4366 616 432 432
query6 316 224 208 208
query7 4621 484 289 289
query8 280 235 229 229
query9 8623 2639 2651 2639
query10 488 327 278 278
query11 15421 14999 14870 14870
query12 158 108 108 108
query13 1659 542 416 416
query14 9224 6088 6068 6068
query15 214 194 173 173
query16 7674 616 471 471
query17 1202 742 607 607
query18 2024 415 323 323
query19 217 198 172 172
query20 132 121 109 109
query21 219 127 103 103
query22 4147 4101 4147 4101
query23 33965 33009 32960 32960
query24 8595 2405 2415 2405
query25 538 464 400 400
query26 907 300 156 156
query27 2683 516 342 342
query28 4305 2157 2100 2100
query29 656 569 459 459
query30 281 224 194 194
query31 908 805 723 723
query32 72 65 64 64
query33 565 357 314 314
query34 808 850 529 529
query35 800 814 724 724
query36 967 1037 915 915
query37 110 102 76 76
query38 4209 4194 4176 4176
query39 1491 1421 1427 1421
query40 220 120 109 109
query41 57 59 53 53
query42 125 108 107 107
query43 517 531 500 500
query44 1305 818 854 818
query45 184 177 168 168
query46 837 1028 652 652
query47 1770 1812 1743 1743
query48 373 437 315 315
query49 683 474 401 401
query50 656 691 402 402
query51 5522 5527 5477 5477
query52 113 110 103 103
query53 218 260 181 181
query54 574 571 515 515
query55 89 81 80 80
query56 315 303 281 281
query57 1198 1202 1147 1147
query58 266 258 273 258
query59 2609 2748 2596 2596
query60 325 328 305 305
query61 137 127 126 126
query62 793 703 670 670
query63 226 198 195 195
query64 3468 1078 677 677
query65 4273 4175 4161 4161
query66 905 416 323 323
query67 15870 15758 15623 15623
query68 8304 894 533 533
query69 483 313 279 279
query70 1190 1169 1140 1140
query71 460 320 307 307
query72 5449 4784 4915 4784
query73 729 691 355 355
query74 9246 9147 8808 8808
query75 3883 3179 2690 2690
query76 3621 1191 726 726
query77 788 379 289 289
query78 9495 9716 8973 8973
query79 1680 834 589 589
query80 640 522 453 453
query81 475 255 221 221
query82 197 128 102 102
query83 285 263 241 241
query84 296 109 93 93
query85 756 408 324 324
query86 325 314 310 310
query87 4472 4470 4394 4394
query88 2942 2314 2306 2306
query89 388 326 291 291
query90 2062 216 219 216
query91 139 155 109 109
query92 79 64 57 57
query93 1108 962 598 598
query94 677 410 310 310
query95 376 294 286 286
query96 492 577 284 284
query97 2710 2742 2701 2701
query98 233 211 207 207
query99 1412 1461 1318 1318
Total cold run time: 272926 ms
Total hot run time: 187418 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3026792132
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3035451138
# FE UT Coverage Report
Increment line coverage ` 83.38% (321/385)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_fb22f5a51e780f7c6e69fd2928b66574465f5027/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_fb22f5a51e780f7c6e69fd2928b66574465f5027/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2986277074
<details>
<summary>ClickBench: <b>Total hot run time: 29.25 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b5320256c6a18d05446c7ed05e53664c535342e, data reload: false
query1 0.03 0.03 0.04
query2 0.07 0.03 0.04
query3 0.24 0.07 0.07
query4 1.62 0.11 0.10
query5 0.46 0.44 0.41
query6 1.17 0.66 0.67
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.58 0.52 0.52
query10 0.59 0.59 0.57
query11 0.15 0.11 0.11
query12 0.16 0.12 0.12
query13 0.63 0.60 0.61
query14 0.82 0.80 0.81
query15 0.90 0.88 0.86
query16 0.38 0.39 0.39
query17 1.09 1.07 1.07
query18 0.23 0.21 0.22
query19 1.97 1.90 1.86
query20 0.02 0.02 0.01
query21 15.39 0.92 0.53
query22 0.76 1.03 0.67
query23 15.02 1.37 0.67
query24 6.65 2.72 0.45
query25 0.31 0.08 0.16
query26 0.67 0.16 0.15
query27 0.07 0.06 0.05
query28 9.44 0.89 0.44
query29 12.59 3.98 3.27
query30 0.24 0.09 0.07
query31 2.83 0.60 0.40
query32 3.24 0.55 0.46
query33 3.11 3.07 3.19
query34 16.08 5.41 4.78
query35 4.87 4.84 4.85
query36 0.71 0.52 0.49
query37 0.09 0.07 0.07
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.18 0.14 0.14
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 103.66 s
Total hot run time: 29.25 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3053136340
<details>
<summary>ClickBench: <b>Total hot run time: 29.15 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0ac0a854575f8810270790b6a58154b043b9529c, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.25 0.07 0.07
query4 1.63 0.11 0.11
query5 0.42 0.44 0.41
query6 1.18 0.66 0.66
query7 0.02 0.02 0.01
query8 0.04 0.03 0.04
query9 0.61 0.52 0.51
query10 0.59 0.58 0.57
query11 0.16 0.11 0.11
query12 0.15 0.12 0.11
query13 0.63 0.61 0.63
query14 0.78 0.83 0.82
query15 0.92 0.86 0.88
query16 0.40 0.38 0.40
query17 1.08 1.06 1.06
query18 0.23 0.22 0.21
query19 1.93 1.82 1.95
query20 0.01 0.01 0.02
query21 15.37 0.87 0.56
query22 0.78 1.19 0.67
query23 14.93 1.38 0.64
query24 7.33 1.29 0.35
query25 0.41 0.33 0.12
query26 0.66 0.17 0.13
query27 0.05 0.06 0.05
query28 9.24 0.95 0.45
query29 12.54 4.01 3.31
query30 0.26 0.09 0.07
query31 2.83 0.60 0.39
query32 3.27 0.56 0.49
query33 3.14 3.25 3.15
query34 16.05 5.41 4.73
query35 4.83 4.84 4.79
query36 0.68 0.52 0.49
query37 0.10 0.07 0.07
query38 0.06 0.04 0.03
query39 0.04 0.02 0.03
query40 0.18 0.15 0.14
query41 0.08 0.03 0.02
query42 0.04 0.03 0.02
query43 0.04 0.03 0.02
Total cold run time: 104.06 s
Total hot run time: 29.15 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2194889439
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
Review Comment:
add javadoc style function comment to all functions, add explain the parameters of functions in detail
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
+ List<Expression> conjunctions,
+ boolean useInnerInfer,
+ ImmutableEqualSet<Slot> parentEqualSet,
+ Map<Slot, Literal> parentConstants) {
+ Map<Slot, Literal> newConstants = Maps.newLinkedHashMapWithExpectedSize(parentConstants.size());
+ newConstants.putAll(parentConstants);
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>(parentEqualSet);
+ for (Expression child : conjunctions) {
+ Optional<Pair<Slot, Expression>> equalItem = findValidEqualItem(child);
+ if (!equalItem.isPresent()) {
+ continue;
+ }
+ Slot slot = equalItem.get().first;
+ Expression expr = equalItem.get().second;
+ // for expression `a = 1 and a * 3 = 10`
+ // if it's in LogicalFilter, then we can infer constant relation `a = 1`, then have:
+ // `a = 1 and 3 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
Review Comment:
```suggestion
// `a = 1 and 1 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
+ List<Expression> conjunctions,
+ boolean useInnerInfer,
+ ImmutableEqualSet<Slot> parentEqualSet,
+ Map<Slot, Literal> parentConstants) {
+ Map<Slot, Literal> newConstants = Maps.newLinkedHashMapWithExpectedSize(parentConstants.size());
+ newConstants.putAll(parentConstants);
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>(parentEqualSet);
+ for (Expression child : conjunctions) {
+ Optional<Pair<Slot, Expression>> equalItem = findValidEqualItem(child);
+ if (!equalItem.isPresent()) {
+ continue;
+ }
+ Slot slot = equalItem.get().first;
+ Expression expr = equalItem.get().second;
+ // for expression `a = 1 and a * 3 = 10`
+ // if it's in LogicalFilter, then we can infer constant relation `a = 1`, then have:
+ // `a = 1 and 3 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
+ // but if it's in LogicalProject, then we shouldn't propagate `a=1` to this expression,
+ // because a may be null, then `a=1 and a * 3 = 10` should evaluate to `NULL` in the project.
+ if (!useInnerInfer && (slot.nullable() || expr.nullable())) {
+ continue;
+ }
+ if (expr instanceof Slot) {
+ newEqualSetBuilder.addEqualPair(slot, (Slot) expr);
+ } else if (!addConstant(newConstants, slot, (Literal) expr)) {
+ return Optional.empty();
+ }
+ }
+
+ ImmutableEqualSet<Slot> newEqualSet = newEqualSetBuilder.build();
+ List<Set<Slot>> multiEqualSlots = newEqualSet.calEqualSetList();
+ for (Set<Slot> slots : multiEqualSlots) {
+ Slot slot = slots.stream().filter(newConstants::containsKey).findFirst().orElse(null);
+ if (slot == null) {
+ continue;
+ }
+ Literal value = newConstants.get(slot);
+ for (Slot s : slots) {
+ if (!addConstant(newConstants, s, value)) {
+ return Optional.empty();
+ }
+ }
+ }
+
+ return Optional.of(Pair.of(newEqualSet, newConstants));
+ }
+
+ private boolean addConstant(Map<Slot, Literal> constants, Slot slot, Literal value) {
Review Comment:
explain the result meaning
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
Review Comment:
`== Boolean.FALSE` is safe? why not just `== false` let it unboxing automaticly?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
+ List<Expression> conjunctions,
+ boolean useInnerInfer,
+ ImmutableEqualSet<Slot> parentEqualSet,
+ Map<Slot, Literal> parentConstants) {
+ Map<Slot, Literal> newConstants = Maps.newLinkedHashMapWithExpectedSize(parentConstants.size());
+ newConstants.putAll(parentConstants);
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>(parentEqualSet);
+ for (Expression child : conjunctions) {
+ Optional<Pair<Slot, Expression>> equalItem = findValidEqualItem(child);
+ if (!equalItem.isPresent()) {
+ continue;
+ }
+ Slot slot = equalItem.get().first;
+ Expression expr = equalItem.get().second;
+ // for expression `a = 1 and a * 3 = 10`
+ // if it's in LogicalFilter, then we can infer constant relation `a = 1`, then have:
+ // `a = 1 and 3 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
+ // but if it's in LogicalProject, then we shouldn't propagate `a=1` to this expression,
+ // because a may be null, then `a=1 and a * 3 = 10` should evaluate to `NULL` in the project.
+ if (!useInnerInfer && (slot.nullable() || expr.nullable())) {
+ continue;
+ }
+ if (expr instanceof Slot) {
+ newEqualSetBuilder.addEqualPair(slot, (Slot) expr);
+ } else if (!addConstant(newConstants, slot, (Literal) expr)) {
+ return Optional.empty();
+ }
+ }
+
+ ImmutableEqualSet<Slot> newEqualSet = newEqualSetBuilder.build();
+ List<Set<Slot>> multiEqualSlots = newEqualSet.calEqualSetList();
+ for (Set<Slot> slots : multiEqualSlots) {
+ Slot slot = slots.stream().filter(newConstants::containsKey).findFirst().orElse(null);
+ if (slot == null) {
+ continue;
+ }
+ Literal value = newConstants.get(slot);
+ for (Slot s : slots) {
+ if (!addConstant(newConstants, s, value)) {
+ return Optional.empty();
+ }
+ }
+ }
+
+ return Optional.of(Pair.of(newEqualSet, newConstants));
+ }
+
+ private boolean addConstant(Map<Slot, Literal> constants, Slot slot, Literal value) {
+ Literal existValue = constants.get(slot);
+ if (existValue == null) {
+ constants.put(slot, value);
+ return true;
+ }
+ return value.equals(existValue)
+ || value instanceof ComparableLiteral && existValue instanceof ComparableLiteral
+ && ((ComparableLiteral) value).compareTo((ComparableLiteral) existValue) == 0;
Review Comment:
add parentheses to let the compund predicate clearer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2811760249
<details>
<summary>ClickBench: <b>Total hot run time: 29.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d87aec37767855cd4ec3c2a005b87064810a0710, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.11 0.10
query3 0.25 0.20 0.19
query4 1.59 0.19 0.19
query5 0.60 0.60 0.60
query6 1.17 0.72 0.72
query7 0.03 0.02 0.01
query8 0.04 0.04 0.03
query9 0.58 0.52 0.52
query10 0.57 0.56 0.58
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.62 0.59 0.60
query14 1.15 1.16 1.21
query15 0.88 0.86 0.86
query16 0.38 0.40 0.38
query17 1.04 1.05 1.03
query18 0.22 0.19 0.19
query19 1.90 1.83 1.79
query20 0.01 0.01 0.02
query21 15.39 0.88 0.55
query22 0.76 1.18 0.63
query23 14.97 1.40 0.65
query24 6.76 1.74 0.89
query25 0.48 0.07 0.15
query26 0.68 0.16 0.14
query27 0.06 0.06 0.05
query28 9.83 0.89 0.46
query29 12.51 3.97 3.30
query30 0.25 0.08 0.07
query31 2.84 0.60 0.38
query32 3.22 0.54 0.49
query33 2.99 2.98 3.12
query34 15.79 5.15 4.51
query35 4.54 4.49 4.55
query36 0.66 0.50 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.13
query41 0.08 0.03 0.02
query42 0.03 0.03 0.02
query43 0.04 0.03 0.02
Total cold run time: 103.71 s
Total hot run time: 29.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2811678575
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823902616
<details>
<summary>TPC-H: <b>Total hot run time: 33970 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 380ba6cacf44785c79fbf8a3313cb0f6d054a66e, data reload: false
------ Round 1 ----------------------------------
q1 26146 5065 5019 5019
q2 2069 284 193 193
q3 10376 1240 701 701
q4 10228 1040 537 537
q5 7555 2352 2310 2310
q6 185 165 132 132
q7 919 771 619 619
q8 9311 1285 1123 1123
q9 6885 5112 5137 5112
q10 6809 2317 1876 1876
q11 466 299 266 266
q12 345 356 227 227
q13 17756 3647 3060 3060
q14 226 225 211 211
q15 544 494 485 485
q16 447 444 393 393
q17 599 862 354 354
q18 7654 7115 7132 7115
q19 1224 948 562 562
q20 333 365 228 228
q21 4044 3358 2473 2473
q22 1049 998 974 974
Total cold run time: 115170 ms
Total hot run time: 33970 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5065 5014 5079 5014
q2 236 333 245 245
q3 2160 2668 2260 2260
q4 1413 1842 1406 1406
q5 4548 4484 4411 4411
q6 207 164 126 126
q7 2030 1925 1695 1695
q8 2581 2543 2525 2525
q9 7231 7105 7227 7105
q10 2971 3181 2738 2738
q11 559 499 483 483
q12 710 779 630 630
q13 3447 3855 3251 3251
q14 276 306 271 271
q15 510 471 491 471
q16 463 519 481 481
q17 1143 1467 1444 1444
q18 7738 7597 7558 7558
q19 815 798 804 798
q20 1984 2017 1890 1890
q21 5096 4644 4615 4615
q22 1049 1002 977 977
Total cold run time: 52232 ms
Total hot run time: 50394 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823040599
<details>
<summary>TPC-H: <b>Total hot run time: 33940 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb7fcae3e6afa2a5aa88d0278ff4850df770d248, data reload: false
------ Round 1 ----------------------------------
q1 26150 5061 5042 5042
q2 2089 286 182 182
q3 10374 1187 682 682
q4 10228 1020 537 537
q5 7541 2366 2302 2302
q6 181 160 133 133
q7 900 744 603 603
q8 9312 1251 1106 1106
q9 6773 5084 5065 5065
q10 6826 2275 1888 1888
q11 461 284 278 278
q12 346 352 227 227
q13 17785 3644 3080 3080
q14 227 226 210 210
q15 545 507 494 494
q16 460 458 403 403
q17 605 841 376 376
q18 7566 7159 7114 7114
q19 1236 970 556 556
q20 335 341 235 235
q21 4119 3403 2455 2455
q22 1038 990 972 972
Total cold run time: 115097 ms
Total hot run time: 33940 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5128 5113 5152 5113
q2 241 335 226 226
q3 2137 2636 2290 2290
q4 1401 1815 1421 1421
q5 4548 4437 4407 4407
q6 218 163 125 125
q7 1921 1865 1743 1743
q8 2604 2486 2460 2460
q9 7142 7067 7082 7067
q10 3015 3159 2758 2758
q11 564 502 490 490
q12 674 788 624 624
q13 3527 3930 3226 3226
q14 292 308 267 267
q15 529 485 489 485
q16 462 504 464 464
q17 1158 1529 1385 1385
q18 7653 7470 7253 7253
q19 801 789 820 789
q20 2022 2066 1872 1872
q21 5340 4682 4559 4559
q22 1073 1036 969 969
Total cold run time: 52450 ms
Total hot run time: 49993 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2879175992
<details>
<summary>TPC-H: <b>Total hot run time: 33724 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c17efe382987bbb5fe265707273ae1c879a3a35, data reload: false
------ Round 1 ----------------------------------
q1 26216 5089 5007 5007
q2 2064 263 178 178
q3 10412 1260 709 709
q4 10226 983 517 517
q5 7528 2346 2364 2346
q6 182 163 132 132
q7 904 750 614 614
q8 9528 1282 1065 1065
q9 6922 5061 5015 5015
q10 6868 2330 1901 1901
q11 487 287 266 266
q12 348 371 218 218
q13 17765 3671 3067 3067
q14 227 228 221 221
q15 548 497 488 488
q16 416 423 391 391
q17 584 862 350 350
q18 7518 7184 7060 7060
q19 1604 978 568 568
q20 329 337 223 223
q21 4000 3295 2427 2427
q22 1025 1005 961 961
Total cold run time: 115701 ms
Total hot run time: 33724 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5203 5031 5059 5031
q2 245 326 232 232
q3 2138 2692 2348 2348
q4 1363 1848 1400 1400
q5 4462 4413 4413 4413
q6 221 169 130 130
q7 2008 1951 1772 1772
q8 2565 2644 2581 2581
q9 7165 7168 6955 6955
q10 3019 3218 2802 2802
q11 576 522 490 490
q12 704 780 631 631
q13 3598 3954 3298 3298
q14 295 294 304 294
q15 555 493 493 493
q16 443 518 447 447
q17 1134 1533 1384 1384
q18 7854 7571 7456 7456
q19 779 729 850 729
q20 2032 2025 1870 1870
q21 5121 4844 4656 4656
q22 1105 1047 1012 1012
Total cold run time: 52585 ms
Total hot run time: 50424 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2889952144
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890628090
<details>
<summary>TPC-H: <b>Total hot run time: 33629 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ee31a1ae3d3e603ceadb2c81ce12e1aec375f8da, data reload: false
------ Round 1 ----------------------------------
q1 26262 4947 4923 4923
q2 2075 300 185 185
q3 10526 1228 694 694
q4 10233 998 501 501
q5 7732 2370 2329 2329
q6 181 163 132 132
q7 913 757 614 614
q8 9331 1227 1069 1069
q9 6833 5074 5077 5074
q10 6868 2296 1885 1885
q11 498 295 280 280
q12 348 342 213 213
q13 17786 3651 3071 3071
q14 232 247 213 213
q15 530 489 491 489
q16 426 435 388 388
q17 612 834 396 396
q18 7672 7187 7136 7136
q19 1993 963 522 522
q20 336 334 225 225
q21 3660 2572 2345 2345
q22 1022 1030 945 945
Total cold run time: 116069 ms
Total hot run time: 33629 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5163 4974 4960 4960
q2 231 330 232 232
q3 2126 2637 2254 2254
q4 1327 1753 1365 1365
q5 4471 4366 4388 4366
q6 226 169 124 124
q7 1967 1927 1752 1752
q8 2561 2500 2519 2500
q9 7228 7149 6932 6932
q10 3005 3224 2762 2762
q11 568 521 517 517
q12 667 802 644 644
q13 3852 3891 3316 3316
q14 297 305 289 289
q15 527 492 487 487
q16 472 486 441 441
q17 1118 1546 1368 1368
q18 7888 7674 7377 7377
q19 793 785 920 785
q20 1962 2055 1922 1922
q21 4653 4476 4518 4476
q22 1082 1078 1026 1026
Total cold run time: 52184 ms
Total hot run time: 49895 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2086006645
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -206,7 +213,53 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
+ ExpressionRewriteContext context = new ExpressionRewriteContext(tempCascadeContext);
+ for (int i = 0; i < getOutputs().size(); i++) {
+ Optional<Literal> value = Optional.empty();
+ if (!constantExprsList.isEmpty()) {
+ value = ExpressionUtils.checkConstantExpr(constantExprsList.get(0).get(i), context);
+ if (!value.isPresent()) {
+ continue;
+ }
+ final int fi = i;
+ Literal literal = value.get();
+ if (constantExprsList.stream()
+ .map(exprs -> ExpressionUtils.checkConstantExpr(exprs.get(fi), context))
+ .anyMatch(val -> !val.isPresent() || !val.get().equals(literal))) {
+ continue;
+ }
+ }
+ for (int childIdx = 0; childIdx < children.size(); childIdx++) {
+ // TODO: use originOutputs = child(childIdx).getOutput() ?
+ List<? extends Slot> originOutputs = regularChildrenOutputs.get(childIdx);
+ Slot slot = originOutputs.get(i);
+ Optional<Expression> childValue = child(childIdx).getLogicalProperties()
+ .getTrait().getUniformValue(slot);
+ if (childValue == null || !childValue.isPresent() || !childValue.get().isConstant()) {
+ value = Optional.empty();
+ break;
+ }
+ Expression childExpr = childValue.get();
+ if (!childExpr.isLiteral()) {
+ childExpr = FoldConstantRule.evaluate(childExpr, context);
+ if (!childExpr.isLiteral()) {
+ continue;
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2829839296
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2834842585
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838692865
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2827524013
<details>
<summary>TPC-H: <b>Total hot run time: 34084 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bac9e2e46a55da229928c9bdb42e58fb56761025, data reload: false
------ Round 1 ----------------------------------
q1 26227 5064 5017 5017
q2 2089 289 187 187
q3 10390 1252 713 713
q4 10225 1035 531 531
q5 7536 2302 2362 2302
q6 185 161 131 131
q7 914 737 619 619
q8 9324 1264 1072 1072
q9 6735 5117 5121 5117
q10 6867 2327 1893 1893
q11 490 276 279 276
q12 368 352 219 219
q13 17777 3666 3099 3099
q14 227 237 214 214
q15 542 511 493 493
q16 448 456 404 404
q17 588 855 363 363
q18 7545 7391 7218 7218
q19 1688 950 536 536
q20 322 327 219 219
q21 3970 3418 2481 2481
q22 1102 1024 980 980
Total cold run time: 115559 ms
Total hot run time: 34084 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5151 5041 5071 5041
q2 231 328 232 232
q3 2154 2600 2299 2299
q4 1457 1862 1431 1431
q5 4486 4403 4417 4403
q6 214 180 133 133
q7 1966 1882 1725 1725
q8 2602 2569 2527 2527
q9 7231 7257 6929 6929
q10 3023 3185 2738 2738
q11 564 528 479 479
q12 683 746 620 620
q13 3516 3861 3317 3317
q14 283 302 271 271
q15 544 493 485 485
q16 487 497 455 455
q17 1138 1490 1416 1416
q18 7820 7688 7460 7460
q19 809 843 950 843
q20 2025 2030 1813 1813
q21 5243 4974 4954 4954
q22 1087 1049 1050 1049
Total cold run time: 52714 ms
Total hot run time: 50620 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3100675572
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2214779477
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalProject.java:
##########
@@ -121,9 +122,13 @@ public String shapeInfo() {
&& context.getSessionVariable().getDetailShapePlanNodesSet().contains(getClass().getSimpleName())) {
StringBuilder builder = new StringBuilder();
builder.append(getClass().getSimpleName());
- // the internal project list's order may be unstable, especial for join tables,
- // so sort the projects to make it stable
- builder.append(projects.stream().map(Expression::shapeInfo).sorted()
+ Stream<String> projectStream = projects.stream().map(Expression::shapeInfo);
+ if (!context.getSessionVariable().enableProjectShapePositionOrder) {
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209046502
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
Review Comment:
fix
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java:
##########
@@ -99,11 +100,15 @@ public static ConnectContext createBasicMvContext(@Nullable ConnectContext paren
ctx.getSessionVariable().showHiddenColumns = false;
ctx.getSessionVariable().allowModifyMaterializedViewData = true;
// Disable add default limit rule to avoid refresh data wrong
Review Comment:
update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3079364039
# FE UT Coverage Report
Increment line coverage ` 83.21% (342/411)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_a20e730c75a1273b1167641933d055aea6470f95/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_a20e730c75a1273b1167641933d055aea6470f95/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2944070906
<details>
<summary>TPC-DS: <b>Total hot run time: 192928 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c0c5557ef07d1fbe8f42072b38acc17bfaed12bd, data reload: false
query1 1420 1079 1104 1079
query2 6171 1841 1861 1841
query3 11171 4774 4661 4661
query4 24991 23909 23428 23428
query5 4404 639 480 480
query6 315 205 201 201
query7 4011 519 300 300
query8 270 234 211 211
query9 8499 2683 2681 2681
query10 448 329 280 280
query11 15829 15134 14863 14863
query12 165 107 109 107
query13 1577 558 438 438
query14 9191 6291 6402 6291
query15 215 209 172 172
query16 7305 677 498 498
query17 1211 782 631 631
query18 1988 428 333 333
query19 207 208 184 184
query20 132 134 132 132
query21 207 129 118 118
query22 4453 4593 4324 4324
query23 34578 33930 33489 33489
query24 8580 2436 2476 2436
query25 522 491 410 410
query26 733 281 160 160
query27 2943 522 365 365
query28 4752 2181 2182 2181
query29 684 599 473 473
query30 274 234 203 203
query31 867 837 749 749
query32 75 69 66 66
query33 534 387 319 319
query34 819 883 565 565
query35 816 838 755 755
query36 963 988 925 925
query37 131 101 78 78
query38 4279 4324 4170 4170
query39 1514 1503 1499 1499
query40 221 128 116 116
query41 66 61 58 58
query42 120 111 113 111
query43 513 531 479 479
query44 1382 838 840 838
query45 188 185 181 181
query46 892 1041 682 682
query47 1858 1905 1787 1787
query48 415 440 322 322
query49 692 525 430 430
query50 694 743 422 422
query51 4192 4252 4271 4252
query52 119 123 106 106
query53 231 286 200 200
query54 605 603 522 522
query55 87 90 92 90
query56 311 308 313 308
query57 1152 1211 1151 1151
query58 277 270 275 270
query59 2674 2702 2601 2601
query60 349 347 340 340
query61 160 156 162 156
query62 805 735 685 685
query63 234 204 206 204
query64 3071 1069 702 702
query65 4378 4288 4326 4288
query66 1011 406 305 305
query67 15994 15588 15521 15521
query68 9063 871 545 545
query69 504 317 283 283
query70 1219 1126 1119 1119
query71 482 334 327 327
query72 5698 4773 5013 4773
query73 745 663 368 368
query74 9021 9195 8673 8673
query75 4171 3220 2686 2686
query76 3678 1207 793 793
query77 788 392 290 290
query78 9407 9486 8692 8692
query79 2104 869 578 578
query80 588 527 476 476
query81 489 257 221 221
query82 452 131 152 131
query83 265 256 232 232
query84 250 108 92 92
query85 768 324 300 300
query86 375 324 292 292
query87 4381 4430 4276 4276
query88 3727 2312 2262 2262
query89 395 314 305 305
query90 1836 220 212 212
query91 147 153 120 120
query92 83 65 60 60
query93 1676 946 585 585
query94 678 436 312 312
query95 375 303 290 290
query96 504 573 291 291
query97 2691 2750 2654 2654
query98 235 228 206 206
query99 1411 1410 1261 1261
Total cold run time: 278141 ms
Total hot run time: 192928 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2952568829
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2900509307
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2903836747
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823854930
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2822977849
<details>
<summary>ClickBench: <b>Total hot run time: 29.52 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51d565f4fc52c0b7e4d1370e096c79be06110bb0, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.10 0.12
query3 0.25 0.20 0.20
query4 1.60 0.20 0.20
query5 0.59 0.59 0.59
query6 1.20 0.72 0.71
query7 0.03 0.02 0.02
query8 0.04 0.04 0.03
query9 0.57 0.52 0.52
query10 0.56 0.59 0.56
query11 0.16 0.11 0.11
query12 0.15 0.11 0.11
query13 0.61 0.59 0.60
query14 1.15 1.22 1.20
query15 0.88 0.86 0.85
query16 0.38 0.38 0.38
query17 1.04 1.06 1.00
query18 0.22 0.20 0.20
query19 1.95 1.86 1.79
query20 0.01 0.02 0.01
query21 15.41 0.93 0.56
query22 0.76 1.15 0.61
query23 15.03 1.41 0.62
query24 7.43 1.46 0.79
query25 0.49 0.13 0.16
query26 0.64 0.17 0.16
query27 0.05 0.05 0.04
query28 9.80 0.84 0.44
query29 12.58 4.04 3.29
query30 0.26 0.09 0.07
query31 2.82 0.60 0.40
query32 3.24 0.55 0.46
query33 3.02 3.02 3.05
query34 15.72 5.21 4.44
query35 4.54 4.54 4.49
query36 0.67 0.50 0.48
query37 0.09 0.06 0.06
query38 0.06 0.04 0.03
query39 0.03 0.02 0.03
query40 0.18 0.14 0.13
query41 0.09 0.03 0.03
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 104.53 s
Total hot run time: 29.52 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2822899904
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2898531210
<details>
<summary>TPC-DS: <b>Total hot run time: 192228 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8398b2cd745248ad60a0a654105ed3b53b02bf7b, data reload: false
query1 1407 1125 1090 1090
query2 6033 1768 1815 1768
query3 11015 4491 4374 4374
query4 55210 25178 23433 23433
query5 5035 536 448 448
query6 324 206 202 202
query7 4882 524 302 302
query8 328 270 252 252
query9 5422 2619 2627 2619
query10 449 329 271 271
query11 15322 15100 15010 15010
query12 161 108 107 107
query13 1035 530 430 430
query14 10187 6215 6334 6215
query15 212 207 181 181
query16 7144 655 496 496
query17 1104 753 621 621
query18 1642 436 330 330
query19 214 209 176 176
query20 128 136 127 127
query21 226 130 111 111
query22 4595 4446 4344 4344
query23 34329 33565 33669 33565
query24 6549 2422 2471 2422
query25 461 477 425 425
query26 718 294 164 164
query27 2282 516 353 353
query28 3328 2172 2152 2152
query29 610 596 439 439
query30 293 245 191 191
query31 845 825 729 729
query32 72 65 68 65
query33 451 384 323 323
query34 773 878 546 546
query35 784 873 751 751
query36 995 1029 909 909
query37 108 101 80 80
query38 4315 4247 4173 4173
query39 1510 1474 1468 1468
query40 206 122 110 110
query41 63 57 55 55
query42 129 107 111 107
query43 514 512 488 488
query44 1350 830 845 830
query45 218 193 173 173
query46 877 1054 658 658
query47 1870 1873 1843 1843
query48 416 449 328 328
query49 707 546 481 481
query50 648 702 415 415
query51 4188 4163 4190 4163
query52 120 116 103 103
query53 227 252 187 187
query54 589 592 515 515
query55 83 86 87 86
query56 301 342 296 296
query57 1168 1187 1129 1129
query58 266 274 261 261
query59 2713 2798 2665 2665
query60 362 332 338 332
query61 148 162 129 129
query62 725 718 656 656
query63 222 182 189 182
query64 1457 1021 681 681
query65 4315 4276 4249 4249
query66 705 402 304 304
query67 16093 16045 15555 15555
query68 7996 878 519 519
query69 563 303 271 271
query70 1211 1110 1110 1110
query71 498 332 312 312
query72 6018 4852 4951 4852
query73 1522 698 353 353
query74 9295 9173 8664 8664
query75 3863 3175 2734 2734
query76 4240 1214 767 767
query77 629 378 291 291
query78 9494 9401 8647 8647
query79 2363 872 580 580
query80 747 526 460 460
query81 476 255 219 219
query82 256 126 93 93
query83 248 243 227 227
query84 297 111 89 89
query85 769 340 280 280
query86 380 294 324 294
query87 4457 4476 4338 4338
query88 3762 2304 2310 2304
query89 414 312 285 285
query90 1799 209 202 202
query91 173 147 118 118
query92 81 67 53 53
query93 2021 948 569 569
query94 643 401 301 301
query95 368 300 277 277
query96 507 577 283 283
query97 2780 2820 2632 2632
query98 235 206 193 193
query99 1375 1391 1268 1268
Total cold run time: 300658 ms
Total hot run time: 192228 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890080395
<details>
<summary>ClickBench: <b>Total hot run time: 28.65 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 571206ee6a2859721f242331e63007ddc92b9688, data reload: false
query1 0.04 0.03 0.04
query2 0.12 0.10 0.11
query3 0.26 0.19 0.18
query4 1.59 0.19 0.11
query5 0.42 0.40 0.44
query6 1.17 0.67 0.66
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.57 0.53 0.51
query10 0.57 0.57 0.56
query11 0.16 0.10 0.11
query12 0.15 0.11 0.12
query13 0.61 0.60 0.59
query14 0.78 0.80 0.81
query15 0.86 0.85 0.84
query16 0.38 0.39 0.40
query17 1.04 1.00 1.00
query18 0.22 0.21 0.22
query19 1.90 1.79 1.82
query20 0.02 0.01 0.01
query21 15.43 0.88 0.56
query22 0.77 1.32 0.64
query23 14.81 1.37 0.66
query24 7.43 1.13 0.43
query25 0.48 0.15 0.16
query26 0.64 0.16 0.14
query27 0.06 0.05 0.05
query28 9.25 0.90 0.44
query29 12.56 4.01 3.36
query30 0.24 0.09 0.06
query31 2.84 0.60 0.38
query32 3.24 0.54 0.46
query33 3.06 3.12 3.11
query34 15.77 5.06 4.49
query35 4.48 4.51 4.52
query36 0.65 0.50 0.46
query37 0.08 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.14 0.13
query41 0.08 0.03 0.02
query42 0.03 0.03 0.02
query43 0.04 0.03 0.02
Total cold run time: 103.13 s
Total hot run time: 28.65 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2919045694
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2975601592
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2976045845
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833835354
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833900596
<details>
<summary>TPC-H: <b>Total hot run time: 34098 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 125afa5d3078f0fc800b481e5d33c85e0970562e, data reload: false
------ Round 1 ----------------------------------
q1 26038 5129 5086 5086
q2 2077 276 185 185
q3 10399 1243 700 700
q4 10223 1028 517 517
q5 8111 2434 2348 2348
q6 181 172 135 135
q7 940 751 616 616
q8 9336 1308 1076 1076
q9 6860 5093 5182 5093
q10 6858 2313 1900 1900
q11 486 293 272 272
q12 356 356 214 214
q13 17817 3683 3111 3111
q14 250 237 218 218
q15 529 486 487 486
q16 446 460 400 400
q17 588 869 354 354
q18 7697 7179 7290 7179
q19 1572 978 561 561
q20 332 335 223 223
q21 4171 2673 2422 2422
q22 1095 1010 1002 1002
Total cold run time: 116362 ms
Total hot run time: 34098 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5211 5103 5080 5080
q2 234 320 229 229
q3 2156 2621 2261 2261
q4 1354 1796 1436 1436
q5 4459 4419 4441 4419
q6 218 172 125 125
q7 2042 1924 1766 1766
q8 2598 2678 2609 2609
q9 7275 7273 7117 7117
q10 3014 3201 2739 2739
q11 582 498 483 483
q12 646 733 630 630
q13 3593 3917 3330 3330
q14 276 322 300 300
q15 540 502 477 477
q16 456 493 461 461
q17 1184 1567 1413 1413
q18 7761 7595 7531 7531
q19 809 871 940 871
q20 1982 1995 1825 1825
q21 5196 4722 4835 4722
q22 1142 1080 1029 1029
Total cold run time: 52728 ms
Total hot run time: 50853 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833428634
<details>
<summary>ClickBench: <b>Total hot run time: 29.67 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 623b77512221f6c2e11c5e755e77011eb36e2a01, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.11
query3 0.25 0.20 0.20
query4 1.59 0.19 0.16
query5 0.59 0.58 0.60
query6 1.20 0.71 0.73
query7 0.03 0.02 0.02
query8 0.04 0.04 0.03
query9 0.58 0.52 0.50
query10 0.59 0.59 0.57
query11 0.16 0.11 0.10
query12 0.14 0.11 0.11
query13 0.60 0.59 0.60
query14 1.14 1.19 1.23
query15 0.86 0.86 0.85
query16 0.38 0.39 0.39
query17 1.03 1.04 1.06
query18 0.21 0.20 0.19
query19 1.99 1.79 1.81
query20 0.01 0.01 0.01
query21 15.40 0.94 0.58
query22 0.76 1.13 0.68
query23 15.00 1.36 0.61
query24 7.26 0.82 1.06
query25 0.50 0.13 0.10
query26 0.71 0.17 0.14
query27 0.05 0.05 0.05
query28 9.13 0.92 0.42
query29 12.59 3.91 3.28
query30 0.26 0.09 0.07
query31 2.81 0.58 0.40
query32 3.22 0.54 0.47
query33 3.07 3.05 3.06
query34 15.63 5.19 4.50
query35 4.53 4.51 4.55
query36 0.66 0.50 0.50
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.17 0.14 0.14
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 103.63 s
Total hot run time: 29.67 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2834857622
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2811737642
<details>
<summary>TPC-H: <b>Total hot run time: 33721 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d87aec37767855cd4ec3c2a005b87064810a0710, data reload: false
------ Round 1 ----------------------------------
q1 26426 5034 5175 5034
q2 2079 283 193 193
q3 10377 1222 717 717
q4 10236 1002 544 544
q5 7557 2370 2308 2308
q6 184 163 130 130
q7 944 750 619 619
q8 9319 1303 1103 1103
q9 6893 5010 5152 5010
q10 6842 2299 1899 1899
q11 503 283 270 270
q12 358 349 214 214
q13 17788 3676 3056 3056
q14 227 223 210 210
q15 533 482 497 482
q16 457 451 400 400
q17 583 868 350 350
q18 7629 7137 7034 7034
q19 1758 947 539 539
q20 321 326 226 226
q21 3814 3318 2409 2409
q22 1036 1004 974 974
Total cold run time: 115864 ms
Total hot run time: 33721 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5198 5176 5023 5023
q2 234 319 231 231
q3 2140 2663 2305 2305
q4 1412 1801 1394 1394
q5 4456 4373 4390 4373
q6 215 169 129 129
q7 1944 1908 1800 1800
q8 2616 2498 2497 2497
q9 7267 7265 6973 6973
q10 3010 3181 2738 2738
q11 585 518 500 500
q12 698 781 598 598
q13 3464 3908 3280 3280
q14 294 300 269 269
q15 522 481 499 481
q16 480 510 460 460
q17 1157 1556 1413 1413
q18 7834 7658 7483 7483
q19 803 854 932 854
q20 1997 1999 1802 1802
q21 5129 4802 4845 4802
q22 1103 1064 1037 1037
Total cold run time: 52558 ms
Total hot run time: 50442 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2872349777
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2875188073
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #49982:
URL: https://github.com/apache/doris/pull/49982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3116327426
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3078838116
<details>
<summary>TPC-DS: <b>Total hot run time: 187555 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7dd3d7e2d7f78983d969a79698664d3686e15112, data reload: false
query1 1011 413 407 407
query2 6522 1745 1773 1745
query3 6745 220 212 212
query4 26227 23329 23557 23329
query5 4323 587 451 451
query6 294 204 208 204
query7 4624 501 285 285
query8 259 221 218 218
query9 8621 2654 2642 2642
query10 464 332 267 267
query11 15389 15188 14817 14817
query12 159 109 102 102
query13 1648 539 404 404
query14 8601 5869 5895 5869
query15 206 203 168 168
query16 7138 624 489 489
query17 946 724 592 592
query18 1994 417 320 320
query19 199 192 167 167
query20 133 118 122 118
query21 215 127 109 109
query22 4611 4673 4761 4673
query23 35139 34130 33291 33291
query24 8389 2405 2354 2354
query25 520 433 411 411
query26 1233 264 150 150
query27 2713 502 339 339
query28 4259 2139 2103 2103
query29 780 552 426 426
query30 315 219 182 182
query31 885 802 736 736
query32 71 65 66 65
query33 536 363 310 310
query34 786 843 540 540
query35 810 808 715 715
query36 969 998 887 887
query37 110 93 73 73
query38 4226 4199 4044 4044
query39 1475 1459 1457 1457
query40 201 112 105 105
query41 55 60 51 51
query42 122 105 107 105
query43 507 502 504 502
query44 1310 824 819 819
query45 182 169 161 161
query46 839 1026 622 622
query47 1802 1812 1712 1712
query48 397 425 303 303
query49 726 496 365 365
query50 629 687 408 408
query51 5395 5479 5485 5479
query52 117 105 94 94
query53 223 250 184 184
query54 567 563 500 500
query55 81 80 81 80
query56 317 289 289 289
query57 1168 1197 1130 1130
query58 273 256 259 256
query59 2564 2749 2555 2555
query60 328 324 317 317
query61 131 126 129 126
query62 795 709 670 670
query63 223 187 183 183
query64 4334 1005 701 701
query65 4316 4223 4180 4180
query66 1138 396 302 302
query67 16250 15895 15570 15570
query68 9044 910 526 526
query69 475 299 282 282
query70 1273 1152 1147 1147
query71 466 335 290 290
query72 5308 4740 4838 4740
query73 699 602 355 355
query74 9058 9176 9160 9160
query75 4207 3199 2761 2761
query76 3638 1149 729 729
query77 838 406 286 286
query78 9261 9513 8735 8735
query79 1387 855 572 572
query80 588 520 456 456
query81 473 260 227 227
query82 191 123 94 94
query83 278 246 238 238
query84 301 169 79 79
query85 744 352 307 307
query86 332 309 299 299
query87 4424 4452 4332 4332
query88 2933 2263 2257 2257
query89 380 325 281 281
query90 2067 207 219 207
query91 138 141 107 107
query92 73 59 61 59
query93 1081 960 593 593
query94 666 408 303 303
query95 379 292 279 279
query96 509 578 281 281
query97 2706 2781 2658 2658
query98 252 201 209 201
query99 1396 1403 1274 1274
Total cold run time: 274100 ms
Total hot run time: 187555 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3027195007
<details>
<summary>TPC-DS: <b>Total hot run time: 181517 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce8c338875395acc46f8f4db2550a6b4de478f44, data reload: false
query1 1023 390 396 390
query2 6539 1670 1692 1670
query3 6748 214 208 208
query4 26946 23535 23313 23313
query5 4340 580 439 439
query6 307 222 202 202
query7 4633 487 287 287
query8 278 223 219 219
query9 8605 2621 2620 2620
query10 475 322 274 274
query11 15728 15075 14789 14789
query12 162 106 105 105
query13 1653 546 404 404
query14 9512 5628 5660 5628
query15 204 188 177 177
query16 7616 622 463 463
query17 1206 734 588 588
query18 2030 419 309 309
query19 190 192 162 162
query20 120 116 116 116
query21 212 125 105 105
query22 4065 4186 4073 4073
query23 33994 33202 33147 33147
query24 8530 2347 2385 2347
query25 541 477 395 395
query26 1225 264 144 144
query27 2772 499 335 335
query28 4325 2134 2106 2106
query29 744 549 436 436
query30 271 215 188 188
query31 910 790 740 740
query32 71 63 62 62
query33 564 380 316 316
query34 784 822 515 515
query35 776 804 735 735
query36 930 966 875 875
query37 104 93 74 74
query38 4091 4002 4050 4002
query39 1521 1434 1469 1434
query40 211 115 107 107
query41 60 58 56 56
query42 119 107 107 107
query43 489 509 473 473
query44 1311 808 808 808
query45 170 168 174 168
query46 831 999 632 632
query47 1717 1792 1716 1716
query48 390 419 312 312
query49 741 487 405 405
query50 626 708 384 384
query51 4110 4112 4055 4055
query52 115 106 101 101
query53 226 252 176 176
query54 584 572 506 506
query55 88 78 80 78
query56 290 290 276 276
query57 1159 1233 1131 1131
query58 267 252 249 249
query59 2592 2607 2523 2523
query60 333 340 303 303
query61 131 130 127 127
query62 811 711 671 671
query63 217 180 186 180
query64 4260 1001 655 655
query65 4273 4130 4166 4130
query66 1072 401 306 306
query67 15810 15597 15393 15393
query68 8419 932 527 527
query69 469 297 271 271
query70 1178 1077 1112 1077
query71 451 322 310 310
query72 5296 2343 5074 2343
query73 783 761 355 355
query74 9098 9130 8700 8700
query75 3879 3206 2653 2653
query76 3694 1139 703 703
query77 775 382 287 287
query78 9313 9629 8714 8714
query79 1619 813 587 587
query80 582 514 430 430
query81 472 258 219 219
query82 441 127 95 95
query83 251 243 242 242
query84 243 105 85 85
query85 813 357 392 357
query86 328 289 274 274
query87 4364 4467 4290 4290
query88 2903 2247 2261 2247
query89 384 319 288 288
query90 2032 210 216 210
query91 145 141 114 114
query92 78 60 55 55
query93 1139 974 605 605
query94 799 409 304 304
query95 467 294 288 288
query96 515 572 286 286
query97 2769 2839 2705 2705
query98 225 222 205 205
query99 1354 1385 1271 1271
Total cold run time: 273026 ms
Total hot run time: 181517 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2808280873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3035248302
<details>
<summary>TPC-H: <b>Total hot run time: 33969 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb22f5a51e780f7c6e69fd2928b66574465f5027, data reload: false
------ Round 1 ----------------------------------
q1 17613 5164 5035 5035
q2 1944 282 184 184
q3 10463 1259 738 738
q4 10235 1017 530 530
q5 7646 2364 2369 2364
q6 177 160 126 126
q7 891 742 603 603
q8 9308 1343 1067 1067
q9 6728 5163 5121 5121
q10 6875 2402 1966 1966
q11 479 293 280 280
q12 352 359 214 214
q13 17764 3717 3084 3084
q14 229 228 219 219
q15 559 479 482 479
q16 428 423 385 385
q17 599 852 357 357
q18 7586 7147 7063 7063
q19 1539 962 595 595
q20 328 357 216 216
q21 3746 3164 2374 2374
q22 1049 1041 969 969
Total cold run time: 106538 ms
Total hot run time: 33969 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5116 5142 5120 5120
q2 239 316 236 236
q3 2199 2705 2300 2300
q4 1405 1783 1341 1341
q5 4216 4569 4419 4419
q6 223 172 127 127
q7 2023 1980 1818 1818
q8 2648 2604 2532 2532
q9 7376 7256 7273 7256
q10 3122 3322 2910 2910
q11 606 540 496 496
q12 679 785 624 624
q13 3635 4009 3416 3416
q14 277 326 287 287
q15 518 478 470 470
q16 448 519 462 462
q17 1188 1565 1371 1371
q18 8141 7658 7450 7450
q19 794 802 841 802
q20 1919 1947 1813 1813
q21 4828 4395 4389 4389
q22 1090 1048 992 992
Total cold run time: 52690 ms
Total hot run time: 50631 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3035338651
<details>
<summary>ClickBench: <b>Total hot run time: 29.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb22f5a51e780f7c6e69fd2928b66574465f5027, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.05 0.05
query3 0.25 0.08 0.08
query4 1.62 0.11 0.10
query5 0.44 0.43 0.42
query6 1.19 0.66 0.66
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.61 0.51 0.52
query10 0.58 0.57 0.59
query11 0.16 0.11 0.12
query12 0.16 0.12 0.12
query13 0.64 0.62 0.61
query14 0.82 0.82 0.82
query15 0.92 0.87 0.87
query16 0.42 0.39 0.39
query17 1.07 1.07 1.04
query18 0.23 0.21 0.22
query19 1.99 1.87 1.88
query20 0.02 0.02 0.02
query21 15.36 0.88 0.55
query22 0.78 1.27 0.90
query23 14.73 1.42 0.60
query24 7.89 0.66 0.44
query25 0.31 0.15 0.11
query26 0.67 0.17 0.14
query27 0.06 0.05 0.06
query28 8.89 0.96 0.46
query29 12.54 4.01 3.35
query30 0.25 0.10 0.07
query31 2.82 0.61 0.40
query32 3.25 0.58 0.47
query33 3.04 3.13 3.15
query34 16.15 5.43 4.84
query35 4.89 4.94 4.83
query36 0.70 0.50 0.49
query37 0.10 0.07 0.07
query38 0.05 0.05 0.04
query39 0.04 0.02 0.02
query40 0.18 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.02 0.03
query43 0.05 0.03 0.03
Total cold run time: 104.2 s
Total hot run time: 29.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2822916600
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2827588749
<details>
<summary>ClickBench: <b>Total hot run time: 29.66 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bac9e2e46a55da229928c9bdb42e58fb56761025, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.11
query3 0.25 0.20 0.20
query4 1.59 0.20 0.19
query5 0.60 0.58 0.61
query6 1.19 0.72 0.72
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.56 0.51 0.50
query10 0.57 0.57 0.56
query11 0.15 0.11 0.11
query12 0.15 0.11 0.11
query13 0.62 0.61 0.60
query14 1.14 1.16 1.18
query15 0.89 0.86 0.86
query16 0.38 0.37 0.40
query17 1.01 1.04 1.04
query18 0.21 0.20 0.20
query19 1.89 1.76 1.77
query20 0.02 0.01 0.02
query21 15.40 0.91 0.55
query22 0.76 1.08 0.70
query23 14.97 1.37 0.62
query24 7.15 0.88 0.78
query25 0.46 0.14 0.14
query26 0.61 0.17 0.14
query27 0.05 0.05 0.06
query28 9.81 0.94 0.43
query29 12.56 3.94 3.31
query30 0.25 0.10 0.07
query31 2.83 0.59 0.39
query32 3.24 0.54 0.46
query33 3.09 3.06 3.00
query34 15.83 5.13 4.52
query35 4.56 4.57 4.53
query36 0.65 0.50 0.48
query37 0.09 0.06 0.07
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.14 0.13
query41 0.08 0.04 0.02
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 104.18 s
Total hot run time: 29.66 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823019370
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833377674
<details>
<summary>ClickBench: <b>Total hot run time: 30.1 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 484c6bfb7960914a92b656959003939aa69b130c, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.10 0.11
query3 0.26 0.19 0.20
query4 1.60 0.20 0.19
query5 0.59 0.58 0.58
query6 1.20 0.71 0.75
query7 0.02 0.02 0.02
query8 0.05 0.04 0.03
query9 0.57 0.53 0.51
query10 0.57 0.58 0.56
query11 0.16 0.11 0.10
query12 0.14 0.11 0.11
query13 0.62 0.59 0.60
query14 1.16 1.21 1.16
query15 0.88 0.87 0.85
query16 0.40 0.38 0.39
query17 1.05 1.04 1.00
query18 0.21 0.19 0.20
query19 1.90 1.79 1.87
query20 0.01 0.02 0.01
query21 15.43 0.89 0.55
query22 0.76 1.17 0.77
query23 14.83 1.34 0.62
query24 6.89 1.44 1.34
query25 0.41 0.28 0.07
query26 0.52 0.16 0.16
query27 0.05 0.05 0.05
query28 9.56 0.82 0.45
query29 12.58 3.94 3.28
query30 0.25 0.09 0.06
query31 2.82 0.59 0.38
query32 3.23 0.55 0.46
query33 3.03 3.00 3.12
query34 15.68 5.09 4.46
query35 4.51 4.55 4.54
query36 0.68 0.49 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.14
query41 0.09 0.03 0.02
query42 0.04 0.03 0.02
query43 0.03 0.03 0.03
Total cold run time: 103.28 s
Total hot run time: 30.1 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833911805
<details>
<summary>TPC-DS: <b>Total hot run time: 192756 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 125afa5d3078f0fc800b481e5d33c85e0970562e, data reload: false
query1 1401 1097 1083 1083
query2 6185 1823 1788 1788
query3 11173 4791 4726 4726
query4 25961 23473 23346 23346
query5 4361 663 482 482
query6 335 227 212 212
query7 3995 498 290 290
query8 313 259 260 259
query9 8485 2553 2565 2553
query10 454 336 259 259
query11 15670 14983 15164 14983
query12 163 114 102 102
query13 1663 524 388 388
query14 9765 6154 6246 6154
query15 196 196 168 168
query16 7607 662 477 477
query17 1306 788 632 632
query18 2064 430 329 329
query19 206 206 181 181
query20 129 124 120 120
query21 204 139 111 111
query22 4674 4560 4426 4426
query23 34759 33804 33498 33498
query24 8936 2470 2472 2470
query25 497 474 399 399
query26 1178 277 151 151
query27 2765 529 341 341
query28 4560 2197 2170 2170
query29 685 574 454 454
query30 274 235 191 191
query31 867 798 765 765
query32 77 70 65 65
query33 531 420 320 320
query34 825 881 556 556
query35 804 832 774 774
query36 985 991 899 899
query37 127 110 76 76
query38 4256 4285 4235 4235
query39 1481 1442 1471 1442
query40 219 119 114 114
query41 58 56 52 52
query42 130 109 109 109
query43 522 518 487 487
query44 1382 837 837 837
query45 190 181 168 168
query46 847 1011 669 669
query47 1861 1882 1812 1812
query48 381 431 316 316
query49 775 524 463 463
query50 680 701 401 401
query51 4203 4223 4191 4191
query52 110 111 108 108
query53 230 274 187 187
query54 602 588 536 536
query55 82 80 83 80
query56 333 311 289 289
query57 1187 1201 1156 1156
query58 273 279 270 270
query59 2716 2747 2633 2633
query60 329 348 306 306
query61 129 129 152 129
query62 821 749 661 661
query63 235 194 194 194
query64 3961 1021 725 725
query65 4483 4358 4353 4353
query66 1105 407 313 313
query67 15908 15923 15282 15282
query68 8278 899 514 514
query69 491 307 260 260
query70 1179 1113 1079 1079
query71 475 332 321 321
query72 5732 4766 4756 4756
query73 665 602 362 362
query74 9200 9039 8987 8987
query75 3925 3216 2710 2710
query76 3775 1184 770 770
query77 796 383 291 291
query78 9393 9343 8676 8676
query79 2216 871 562 562
query80 581 510 453 453
query81 494 254 219 219
query82 482 138 99 99
query83 247 245 233 233
query84 251 100 88 88
query85 764 323 282 282
query86 380 314 273 273
query87 4382 4465 4240 4240
query88 3816 2218 2209 2209
query89 412 315 282 282
query90 1877 218 228 218
query91 226 148 111 111
query92 77 62 57 57
query93 1850 940 571 571
query94 671 410 305 305
query95 378 291 296 291
query96 484 556 273 273
query97 3136 3233 3088 3088
query98 244 204 208 204
query99 1330 1414 1256 1256
Total cold run time: 281876 ms
Total hot run time: 192756 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2943852685
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2900673078
<details>
<summary>TPC-DS: <b>Total hot run time: 192031 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd9a107d966f7614132b60ddf52bb5c27d8ae0ff, data reload: false
query1 1454 1095 1068 1068
query2 6418 1904 1853 1853
query3 11048 4559 4438 4438
query4 53515 25684 23803 23803
query5 5152 510 469 469
query6 350 223 204 204
query7 4886 521 290 290
query8 327 257 259 257
query9 5709 2652 2662 2652
query10 466 331 262 262
query11 14957 15017 14787 14787
query12 154 104 106 104
query13 1025 535 408 408
query14 10102 6208 6391 6208
query15 221 198 179 179
query16 7016 641 502 502
query17 1077 772 624 624
query18 1522 397 314 314
query19 199 190 159 159
query20 138 134 135 134
query21 205 135 115 115
query22 4397 4377 4248 4248
query23 34477 33483 33277 33277
query24 6622 2412 2412 2412
query25 462 490 428 428
query26 683 282 158 158
query27 2202 526 348 348
query28 2986 2222 2187 2187
query29 604 581 452 452
query30 283 226 193 193
query31 829 813 740 740
query32 70 64 60 60
query33 468 358 300 300
query34 790 850 537 537
query35 788 825 765 765
query36 1024 1016 927 927
query37 118 109 82 82
query38 4184 4324 4211 4211
query39 1525 1476 1459 1459
query40 214 114 113 113
query41 59 54 52 52
query42 130 113 112 112
query43 524 518 489 489
query44 1351 848 826 826
query45 186 179 172 172
query46 880 1052 648 648
query47 1861 1928 1779 1779
query48 427 458 333 333
query49 716 572 531 531
query50 674 715 416 416
query51 4220 4279 4228 4228
query52 115 112 102 102
query53 233 267 195 195
query54 583 601 537 537
query55 91 89 83 83
query56 349 312 293 293
query57 1171 1188 1118 1118
query58 269 271 262 262
query59 2667 2745 2693 2693
query60 350 338 323 323
query61 157 154 157 154
query62 705 742 711 711
query63 234 195 198 195
query64 1491 1076 726 726
query65 4452 4206 4232 4206
query66 714 401 310 310
query67 16005 15477 15268 15268
query68 5537 891 513 513
query69 519 313 266 266
query70 1165 1098 1087 1087
query71 453 317 296 296
query72 5942 4916 4824 4824
query73 1294 637 357 357
query74 8918 9205 8677 8677
query75 3522 3292 2682 2682
query76 3730 1205 770 770
query77 559 392 291 291
query78 9943 9607 8608 8608
query79 2160 864 581 581
query80 816 532 454 454
query81 505 256 220 220
query82 413 125 94 94
query83 268 249 232 232
query84 306 99 87 87
query85 819 327 300 300
query86 406 301 277 277
query87 4424 4518 4422 4422
query88 3398 2253 2273 2253
query89 406 309 288 288
query90 1887 213 208 208
query91 160 141 114 114
query92 74 61 55 55
query93 2672 954 577 577
query94 666 407 296 296
query95 372 296 277 277
query96 500 575 279 279
query97 2738 2835 2654 2654
query98 239 214 198 198
query99 1318 1409 1305 1305
Total cold run time: 295410 ms
Total hot run time: 192031 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2903940217
<details>
<summary>TPC-H: <b>Total hot run time: 33866 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed7ada88affe7e1c21839f38dbc7af718c33487d, data reload: false
------ Round 1 ----------------------------------
q1 26750 5004 4992 4992
q2 2088 315 187 187
q3 10354 1249 683 683
q4 10211 1003 525 525
q5 7524 2351 2348 2348
q6 178 166 130 130
q7 913 744 614 614
q8 9343 1264 1047 1047
q9 6797 5077 5162 5077
q10 6829 2324 1894 1894
q11 486 294 275 275
q12 348 353 213 213
q13 17774 3687 3059 3059
q14 239 229 209 209
q15 540 486 498 486
q16 415 433 385 385
q17 597 860 365 365
q18 7566 7345 7281 7281
q19 1430 962 555 555
q20 338 345 225 225
q21 3751 3172 2344 2344
q22 1046 1035 972 972
Total cold run time: 115517 ms
Total hot run time: 33866 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5029 5069 5040 5040
q2 252 338 235 235
q3 2157 2691 2292 2292
q4 1386 1819 1337 1337
q5 4464 4407 4406 4406
q6 222 174 129 129
q7 2058 1952 1735 1735
q8 2586 2529 2519 2519
q9 7272 7307 7016 7016
q10 3040 3240 2711 2711
q11 572 510 525 510
q12 676 753 637 637
q13 3510 3863 3267 3267
q14 297 311 282 282
q15 530 479 490 479
q16 440 508 440 440
q17 1165 1569 1378 1378
q18 7805 7645 7375 7375
q19 829 826 859 826
q20 2029 2015 1841 1841
q21 4745 4570 4426 4426
q22 1133 1050 1041 1041
Total cold run time: 52197 ms
Total hot run time: 49922 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2903984867
<details>
<summary>ClickBench: <b>Total hot run time: 29.28 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed7ada88affe7e1c21839f38dbc7af718c33487d, data reload: false
query1 0.04 0.03 0.04
query2 0.12 0.11 0.11
query3 0.26 0.18 0.20
query4 1.59 0.20 0.19
query5 0.46 0.45 0.44
query6 1.16 0.67 0.66
query7 0.02 0.02 0.02
query8 0.04 0.04 0.04
query9 0.60 0.52 0.53
query10 0.57 0.57 0.57
query11 0.16 0.10 0.11
query12 0.15 0.12 0.12
query13 0.62 0.60 0.60
query14 0.80 0.81 0.80
query15 0.89 0.88 0.86
query16 0.39 0.39 0.39
query17 1.03 1.03 1.07
query18 0.23 0.21 0.22
query19 1.96 1.84 1.87
query20 0.01 0.01 0.01
query21 15.43 0.88 0.54
query22 0.76 1.21 0.76
query23 14.79 1.42 0.67
query24 7.22 1.19 0.70
query25 0.49 0.08 0.08
query26 0.67 0.17 0.15
query27 0.06 0.05 0.05
query28 9.25 0.87 0.45
query29 12.56 4.12 3.39
query30 0.26 0.10 0.06
query31 2.81 0.59 0.39
query32 3.23 0.55 0.48
query33 3.08 3.07 3.04
query34 15.75 5.13 4.49
query35 4.52 4.49 4.52
query36 0.65 0.49 0.47
query37 0.09 0.07 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.19 0.14 0.13
query41 0.09 0.03 0.03
query42 0.04 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 103.17 s
Total hot run time: 29.28 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2809403081
<details>
<summary>TPC-DS: <b>Total hot run time: 191448 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eaef1200d54db43a8b5ff63aa1975bd8d9b89cfb, data reload: false
query1 1420 1080 1061 1061
query2 6114 1773 1769 1769
query3 10998 4640 4660 4640
query4 26554 23713 23602 23602
query5 5138 617 476 476
query6 325 221 214 214
query7 3993 509 282 282
query8 312 266 243 243
query9 8502 2553 2564 2553
query10 506 317 258 258
query11 15147 15167 14840 14840
query12 159 111 105 105
query13 1554 531 394 394
query14 8835 6141 6224 6141
query15 200 196 170 170
query16 7274 624 471 471
query17 1172 780 581 581
query18 2009 422 315 315
query19 195 196 185 185
query20 131 123 125 123
query21 207 131 108 108
query22 4404 4517 4241 4241
query23 34612 33403 33735 33403
query24 8436 2401 2410 2401
query25 538 465 387 387
query26 1210 278 149 149
query27 2759 522 340 340
query28 4512 2149 2150 2149
query29 724 584 462 462
query30 280 224 196 196
query31 875 824 772 772
query32 77 66 69 66
query33 546 393 333 333
query34 837 886 540 540
query35 843 849 776 776
query36 988 1001 895 895
query37 120 102 80 80
query38 4282 4248 4180 4180
query39 1498 1461 1454 1454
query40 242 120 114 114
query41 63 54 58 54
query42 125 111 106 106
query43 497 503 460 460
query44 1314 849 806 806
query45 185 175 164 164
query46 845 1025 650 650
query47 1849 1833 1807 1807
query48 407 435 310 310
query49 770 538 439 439
query50 681 682 410 410
query51 4271 4429 4259 4259
query52 112 100 99 99
query53 224 258 193 193
query54 582 577 513 513
query55 85 81 97 81
query56 303 307 287 287
query57 1180 1180 1123 1123
query58 273 264 256 256
query59 2732 2702 2626 2626
query60 339 342 313 313
query61 134 147 127 127
query62 790 753 649 649
query63 227 191 185 185
query64 4177 1035 711 711
query65 4447 4351 4368 4351
query66 1135 402 329 329
query67 16383 15401 15320 15320
query68 8259 894 527 527
query69 517 320 281 281
query70 1113 1113 1111 1111
query71 421 320 304 304
query72 5317 4850 4631 4631
query73 628 563 347 347
query74 8900 9152 8671 8671
query75 3398 3176 2711 2711
query76 3465 1191 749 749
query77 702 375 278 278
query78 9014 9201 8378 8378
query79 1679 874 545 545
query80 567 535 433 433
query81 479 266 217 217
query82 198 126 101 101
query83 251 253 322 253
query84 247 101 85 85
query85 706 306 266 266
query86 362 319 283 283
query87 4399 4558 4357 4357
query88 3033 2178 2188 2178
query89 393 313 273 273
query90 1933 202 207 202
query91 141 150 119 119
query92 78 61 59 59
query93 1933 937 564 564
query94 658 414 311 311
query95 366 284 291 284
query96 487 560 274 274
query97 3104 3229 3083 3083
query98 239 213 205 205
query99 1334 1394 1271 1271
Total cold run time: 276979 ms
Total hot run time: 191448 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2986236627
<details>
<summary>TPC-H: <b>Total hot run time: 33953 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b5320256c6a18d05446c7ed05e53664c535342e, data reload: false
------ Round 1 ----------------------------------
q1 17616 5155 5003 5003
q2 1934 287 186 186
q3 10489 1403 709 709
q4 10303 1006 527 527
q5 8086 2316 2324 2316
q6 187 160 133 133
q7 901 739 626 626
q8 9323 1298 1140 1140
q9 6857 5042 5071 5042
q10 6904 2391 1957 1957
q11 493 288 276 276
q12 339 348 217 217
q13 17812 3660 3105 3105
q14 231 236 211 211
q15 574 478 489 478
q16 432 434 392 392
q17 603 855 379 379
q18 7525 7062 7125 7062
q19 2162 987 526 526
q20 328 323 237 237
q21 3584 3193 2486 2486
q22 1010 982 945 945
Total cold run time: 107693 ms
Total hot run time: 33953 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5128 4963 4954 4954
q2 246 323 227 227
q3 2168 2621 2233 2233
q4 1365 1807 1358 1358
q5 4213 4159 4374 4159
q6 207 171 134 134
q7 2027 1894 1823 1823
q8 2566 2616 2458 2458
q9 7134 7128 7031 7031
q10 3110 3224 2811 2811
q11 573 498 488 488
q12 685 775 644 644
q13 3459 3904 3370 3370
q14 287 302 280 280
q15 545 487 479 479
q16 461 478 460 460
q17 1161 1492 1362 1362
q18 7221 7145 6939 6939
q19 748 732 795 732
q20 1906 1964 1825 1825
q21 4789 4374 4362 4362
q22 1044 1022 976 976
Total cold run time: 51043 ms
Total hot run time: 49105 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3078061696
# FE UT Coverage Report
Increment line coverage ` 83.57% (346/414)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_6da27198f400cdcb828fc2d6ad52b1c2028fda62/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_6da27198f400cdcb828fc2d6ad52b1c2028fda62/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3079092564
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3079249426
<details>
<summary>TPC-H: <b>Total hot run time: 33934 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a20e730c75a1273b1167641933d055aea6470f95, data reload: false
------ Round 1 ----------------------------------
q1 17598 5312 5096 5096
q2 1921 276 177 177
q3 10322 1251 718 718
q4 10215 988 557 557
q5 7526 2343 2357 2343
q6 174 156 129 129
q7 884 752 612 612
q8 9323 1230 1080 1080
q9 6770 5052 5086 5052
q10 6897 2443 1968 1968
q11 479 298 270 270
q12 341 346 220 220
q13 17766 3659 3094 3094
q14 224 228 220 220
q15 553 480 485 480
q16 414 435 380 380
q17 574 862 343 343
q18 7568 7212 7090 7090
q19 1206 944 550 550
q20 337 330 220 220
q21 3758 3228 2382 2382
q22 1010 1019 953 953
Total cold run time: 105860 ms
Total hot run time: 33934 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5184 5130 5140 5130
q2 253 314 224 224
q3 2176 2631 2296 2296
q4 1308 1786 1305 1305
q5 4206 4396 4510 4396
q6 220 168 129 129
q7 2075 2012 1815 1815
q8 2567 2600 2511 2511
q9 7345 7275 7356 7275
q10 3101 3286 2875 2875
q11 601 515 485 485
q12 731 810 617 617
q13 3642 4000 3241 3241
q14 332 318 341 318
q15 515 474 456 456
q16 474 486 448 448
q17 1227 1558 1357 1357
q18 7816 7565 7839 7565
q19 817 821 889 821
q20 2039 2023 1879 1879
q21 4838 4271 4391 4271
q22 1076 1039 976 976
Total cold run time: 52543 ms
Total hot run time: 50390 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3117229766
<details>
<summary>ClickBench: <b>Total hot run time: 32.52 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3f0b61e7e8cc8de1b85cb8c4ac561353c78b15d, data reload: false
query1 0.04 0.03 0.04
query2 0.09 0.04 0.04
query3 0.24 0.07 0.08
query4 1.63 0.11 0.10
query5 0.42 0.43 0.40
query6 1.16 0.65 0.67
query7 0.02 0.02 0.02
query8 0.05 0.04 0.04
query9 0.58 0.53 0.52
query10 0.58 0.58 0.58
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.63 0.61 0.60
query14 0.79 0.84 0.83
query15 0.90 0.86 0.87
query16 0.39 0.39 0.38
query17 1.01 1.10 1.05
query18 0.21 0.19 0.19
query19 2.03 1.85 1.92
query20 0.01 0.01 0.01
query21 15.40 0.90 0.54
query22 0.79 1.24 0.83
query23 14.71 1.36 0.67
query24 6.71 1.31 0.48
query25 0.47 0.28 0.07
query26 0.63 0.17 0.14
query27 0.05 0.05 0.05
query28 9.54 0.95 0.42
query29 12.73 3.91 3.31
query30 3.13 3.01 3.01
query31 2.81 0.56 0.38
query32 3.23 0.55 0.47
query33 3.03 3.17 3.16
query34 15.70 5.42 4.82
query35 4.79 4.89 4.91
query36 0.69 0.53 0.48
query37 0.09 0.07 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.02
query40 0.18 0.14 0.14
query41 0.08 0.02 0.03
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 106.01 s
Total hot run time: 32.52 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3116986103
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3027158796
<details>
<summary>TPC-H: <b>Total hot run time: 33829 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce8c338875395acc46f8f4db2550a6b4de478f44, data reload: false
------ Round 1 ----------------------------------
q1 17603 5294 5063 5063
q2 1985 283 180 180
q3 10449 1288 704 704
q4 10251 1017 514 514
q5 7967 2433 2327 2327
q6 195 159 125 125
q7 905 752 589 589
q8 9319 1297 1058 1058
q9 7489 5099 5112 5099
q10 6887 2396 1976 1976
q11 460 294 280 280
q12 344 350 207 207
q13 17765 3675 3168 3168
q14 230 241 217 217
q15 551 490 480 480
q16 432 426 380 380
q17 581 878 349 349
q18 7502 7158 7113 7113
q19 1597 975 561 561
q20 342 336 223 223
q21 3670 2676 2276 2276
q22 1045 998 940 940
Total cold run time: 107569 ms
Total hot run time: 33829 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5558 5069 5057 5057
q2 251 330 218 218
q3 2200 2745 2292 2292
q4 1377 1779 1426 1426
q5 4256 4548 4528 4528
q6 214 170 127 127
q7 1951 2002 1806 1806
q8 2682 2628 2598 2598
q9 7393 7391 7239 7239
q10 3180 3310 2929 2929
q11 571 519 501 501
q12 690 760 692 692
q13 3621 4034 3413 3413
q14 272 326 299 299
q15 533 491 484 484
q16 487 529 455 455
q17 1162 1508 1432 1432
q18 7821 8110 7465 7465
q19 776 744 856 744
q20 1928 2011 1806 1806
q21 4867 4457 4305 4305
q22 1070 1007 971 971
Total cold run time: 52860 ms
Total hot run time: 50787 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3086496191
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209088742
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
Review Comment:
update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076662602
# FE UT Coverage Report
Increment line coverage ` 83.66% (338/404)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_79a5db32079ea3bcb6f30e554e2924fe379da810/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_79a5db32079ea3bcb6f30e554e2924fe379da810/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2220933542
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java:
##########
@@ -259,6 +260,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
bottomUp(new InlineLogicalView())
),
topic("Eliminate optimization",
+ // custom(RuleType.CONSTANT_PROPAGATION, ConstantPropagation::new),
Review Comment:
what this? similar commented rule below, either add comment to explain why comment them temporary or remove them directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3100931348
<details>
<summary>TPC-DS: <b>Total hot run time: 185598 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59fd2685984525c77c64590d5aaa8ec806504fce, data reload: false
query1 989 399 398 398
query2 6509 1716 1661 1661
query3 6757 230 233 230
query4 26797 23216 23536 23216
query5 4347 635 494 494
query6 314 218 214 214
query7 4634 502 290 290
query8 267 236 227 227
query9 8546 2906 2888 2888
query10 447 350 312 312
query11 15843 14974 14853 14853
query12 164 112 112 112
query13 1637 526 407 407
query14 9376 5969 5876 5876
query15 210 191 177 177
query16 7334 649 461 461
query17 1176 695 569 569
query18 1996 407 311 311
query19 189 203 181 181
query20 173 118 114 114
query21 214 120 103 103
query22 4153 4156 4054 4054
query23 33927 33269 33366 33269
query24 8156 2387 2433 2387
query25 551 467 400 400
query26 1244 277 154 154
query27 2757 508 357 357
query28 4335 2170 2124 2124
query29 729 584 451 451
query30 290 223 198 198
query31 911 813 726 726
query32 91 75 83 75
query33 569 384 351 351
query34 820 853 530 530
query35 806 841 754 754
query36 987 1025 934 934
query37 124 101 88 88
query38 4184 4165 4077 4077
query39 1524 1453 1437 1437
query40 222 128 123 123
query41 67 61 62 61
query42 125 116 119 116
query43 501 498 458 458
query44 1366 880 852 852
query45 192 182 169 169
query46 859 1005 651 651
query47 1785 1824 1765 1765
query48 423 431 334 334
query49 734 503 441 441
query50 663 690 407 407
query51 5463 5461 5442 5442
query52 118 115 104 104
query53 281 269 187 187
query54 592 596 520 520
query55 93 85 93 85
query56 306 328 348 328
query57 1193 1193 1144 1144
query58 283 273 295 273
query59 2562 2636 2592 2592
query60 356 337 330 330
query61 126 120 119 119
query62 827 722 652 652
query63 225 195 201 195
query64 4219 987 682 682
query65 4268 4229 4188 4188
query66 1165 411 342 342
query67 15910 15581 15401 15401
query68 8422 911 586 586
query69 485 355 291 291
query70 1238 1117 1149 1117
query71 463 331 309 309
query72 5320 2429 5108 2429
query73 804 757 362 362
query74 8895 9241 9004 9004
query75 3906 3107 2689 2689
query76 3648 1135 730 730
query77 796 396 320 320
query78 9670 9916 8940 8940
query79 2063 790 578 578
query80 644 541 471 471
query81 480 255 228 228
query82 434 139 109 109
query83 251 251 240 240
query84 246 113 91 91
query85 788 361 330 330
query86 347 330 301 301
query87 4458 4475 4439 4439
query88 3146 2308 2309 2308
query89 390 324 289 289
query90 1949 218 219 218
query91 138 143 115 115
query92 85 71 70 70
query93 1140 962 597 597
query94 668 390 286 286
query95 392 318 307 307
query96 541 590 283 283
query97 2737 2726 2652 2652
query98 249 215 215 215
query99 1460 1435 1273 1273
Total cold run time: 275254 ms
Total hot run time: 185598 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3100885744
<details>
<summary>TPC-H: <b>Total hot run time: 34089 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59fd2685984525c77c64590d5aaa8ec806504fce, data reload: false
------ Round 1 ----------------------------------
q1 17561 5484 5103 5103
q2 1959 273 179 179
q3 10344 1292 719 719
q4 10290 984 523 523
q5 8612 2369 2360 2360
q6 198 162 130 130
q7 888 751 619 619
q8 9332 1334 1097 1097
q9 7157 5146 5181 5146
q10 6891 2395 1960 1960
q11 498 281 271 271
q12 339 347 228 228
q13 17770 3706 3095 3095
q14 242 229 213 213
q15 567 485 496 485
q16 429 436 381 381
q17 595 850 359 359
q18 7574 7230 7124 7124
q19 1281 967 544 544
q20 337 339 220 220
q21 4004 3130 2338 2338
q22 1040 1023 995 995
Total cold run time: 107908 ms
Total hot run time: 34089 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5132 5110 5124 5110
q2 244 329 217 217
q3 2154 2689 2292 2292
q4 1370 1748 1361 1361
q5 4394 4452 4433 4433
q6 210 162 130 130
q7 2264 1915 1834 1834
q8 2602 2552 2517 2517
q9 7545 7374 7269 7269
q10 3083 3317 2857 2857
q11 598 522 510 510
q12 702 769 668 668
q13 3619 4151 3423 3423
q14 296 311 280 280
q15 520 473 472 472
q16 424 489 467 467
q17 1236 1648 1376 1376
q18 8073 7676 7609 7609
q19 773 816 892 816
q20 1993 1974 1865 1865
q21 4761 4370 4328 4328
q22 1072 1051 987 987
Total cold run time: 53065 ms
Total hot run time: 50821 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3100947600
<details>
<summary>ClickBench: <b>Total hot run time: 32.72 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59fd2685984525c77c64590d5aaa8ec806504fce, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.24 0.08 0.07
query4 1.62 0.11 0.11
query5 0.45 0.45 0.41
query6 1.20 0.67 0.67
query7 0.03 0.02 0.02
query8 0.04 0.03 0.04
query9 0.59 0.54 0.51
query10 0.59 0.57 0.57
query11 0.15 0.11 0.10
query12 0.14 0.12 0.12
query13 0.62 0.63 0.61
query14 0.81 0.83 0.83
query15 0.90 0.88 0.87
query16 0.39 0.37 0.40
query17 1.06 1.07 1.06
query18 0.23 0.23 0.22
query19 1.98 1.85 1.89
query20 0.01 0.02 0.01
query21 15.38 0.88 0.53
query22 0.76 1.24 0.83
query23 14.82 1.40 0.64
query24 6.73 1.81 0.72
query25 0.54 0.17 0.07
query26 0.59 0.17 0.14
query27 0.06 0.05 0.06
query28 10.44 0.97 0.44
query29 12.56 3.96 3.33
query30 3.09 3.06 3.01
query31 2.83 0.57 0.39
query32 3.24 0.55 0.49
query33 3.16 3.06 3.10
query34 15.73 5.45 4.83
query35 4.90 4.87 4.86
query36 0.69 0.50 0.49
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.04 0.02 0.03
query40 0.17 0.13 0.14
query41 0.08 0.03 0.03
query42 0.03 0.02 0.03
query43 0.03 0.04 0.03
Total cold run time: 107.18 s
Total hot run time: 32.72 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2986251074
<details>
<summary>TPC-DS: <b>Total hot run time: 185182 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b5320256c6a18d05446c7ed05e53664c535342e, data reload: false
query1 1034 382 395 382
query2 6547 1855 1825 1825
query3 6746 228 236 228
query4 26576 23312 23143 23143
query5 4327 637 486 486
query6 321 214 210 210
query7 4638 498 305 305
query8 295 236 231 231
query9 8653 2622 2629 2622
query10 517 353 302 302
query11 15255 15209 14897 14897
query12 161 113 110 110
query13 1667 546 418 418
query14 9417 6250 6241 6241
query15 220 212 166 166
query16 7712 636 446 446
query17 1203 703 571 571
query18 2019 403 321 321
query19 190 187 175 175
query20 123 119 119 119
query21 216 127 108 108
query22 4124 4180 4137 4137
query23 34502 33012 33208 33012
query24 8455 2367 2407 2367
query25 547 468 398 398
query26 1087 270 153 153
query27 2727 513 352 352
query28 4288 2100 2077 2077
query29 715 545 431 431
query30 287 227 202 202
query31 930 842 735 735
query32 81 66 64 64
query33 554 392 314 314
query34 813 869 536 536
query35 795 819 728 728
query36 974 968 867 867
query37 110 108 77 77
query38 4229 4113 4138 4113
query39 1471 1444 1419 1419
query40 215 119 110 110
query41 64 61 59 59
query42 119 109 109 109
query43 498 523 465 465
query44 1359 841 834 834
query45 185 180 167 167
query46 852 1021 621 621
query47 1739 1790 1711 1711
query48 389 417 321 321
query49 755 522 400 400
query50 661 682 417 417
query51 4126 4242 4138 4138
query52 105 115 103 103
query53 232 261 182 182
query54 584 572 499 499
query55 88 86 94 86
query56 306 311 290 290
query57 1183 1182 1133 1133
query58 274 252 285 252
query59 2480 2678 2481 2481
query60 345 324 316 316
query61 142 156 152 152
query62 845 722 699 699
query63 233 197 194 194
query64 4078 1094 670 670
query65 4256 4190 4179 4179
query66 973 414 301 301
query67 15736 15568 15322 15322
query68 8718 920 547 547
query69 493 311 280 280
query70 1191 1134 1125 1125
query71 475 334 288 288
query72 5373 4693 4681 4681
query73 691 616 353 353
query74 9001 9052 8737 8737
query75 4005 3202 2697 2697
query76 3622 1209 782 782
query77 794 398 293 293
query78 9303 9831 8775 8775
query79 1874 780 561 561
query80 593 542 435 435
query81 471 272 234 234
query82 422 133 102 102
query83 260 256 231 231
query84 243 112 94 94
query85 788 346 321 321
query86 342 315 280 280
query87 4436 4382 4281 4281
query88 2966 2304 2296 2296
query89 399 316 282 282
query90 1942 212 211 211
query91 148 136 112 112
query92 82 64 65 64
query93 1151 941 608 608
query94 667 408 283 283
query95 388 299 284 284
query96 490 578 281 281
query97 2705 2760 2687 2687
query98 234 216 237 216
query99 1374 1420 1262 1262
Total cold run time: 272989 ms
Total hot run time: 185182 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3035172404
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795830140
<details>
<summary>TPC-H: <b>Total hot run time: 33911 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f28a750f710b6d3e443dd61dcdbbcede2b504357, data reload: false
------ Round 1 ----------------------------------
q1 26129 5091 4936 4936
q2 2062 285 183 183
q3 10393 1201 686 686
q4 10250 997 528 528
q5 7531 2359 2311 2311
q6 181 166 132 132
q7 910 734 626 626
q8 9337 1256 1075 1075
q9 6824 5111 5037 5037
q10 6797 2287 1908 1908
q11 471 289 279 279
q12 350 351 225 225
q13 17785 3679 3097 3097
q14 224 228 213 213
q15 554 498 498 498
q16 619 619 579 579
q17 593 854 379 379
q18 7542 7025 7170 7025
q19 1257 938 536 536
q20 330 326 218 218
q21 3977 3383 2465 2465
q22 1047 1034 975 975
Total cold run time: 115163 ms
Total hot run time: 33911 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5111 5066 5042 5042
q2 233 318 225 225
q3 2144 2615 2254 2254
q4 1386 1777 1390 1390
q5 4486 4397 4375 4375
q6 214 171 132 132
q7 1957 1908 1741 1741
q8 2593 2665 2555 2555
q9 7299 7241 6990 6990
q10 2995 3159 2746 2746
q11 572 501 518 501
q12 679 759 635 635
q13 3545 3906 3336 3336
q14 275 288 266 266
q15 527 499 464 464
q16 637 676 660 660
q17 1140 1555 1374 1374
q18 7703 7689 7401 7401
q19 801 820 944 820
q20 1985 2022 1851 1851
q21 5117 4805 5013 4805
q22 1114 1049 1056 1049
Total cold run time: 52513 ms
Total hot run time: 50612 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795843603
<details>
<summary>TPC-DS: <b>Total hot run time: 191908 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f28a750f710b6d3e443dd61dcdbbcede2b504357, data reload: false
query1 1403 1070 1036 1036
query2 6115 1950 1891 1891
query3 11007 4435 4376 4376
query4 53065 25286 23337 23337
query5 5301 485 445 445
query6 369 206 211 206
query7 5004 480 275 275
query8 323 286 259 259
query9 6352 2621 2646 2621
query10 447 324 264 264
query11 15015 14969 14779 14779
query12 150 107 109 107
query13 1094 500 370 370
query14 10194 6597 6376 6376
query15 206 188 190 188
query16 7133 682 504 504
query17 1117 753 602 602
query18 1535 398 335 335
query19 206 202 165 165
query20 135 123 119 119
query21 223 128 108 108
query22 4327 4398 4306 4306
query23 34190 33474 33648 33474
query24 6566 2444 2424 2424
query25 488 459 404 404
query26 714 265 152 152
query27 2251 502 335 335
query28 2965 2437 2433 2433
query29 579 563 436 436
query30 282 225 190 190
query31 827 818 758 758
query32 74 66 67 66
query33 473 392 322 322
query34 833 859 517 517
query35 835 877 761 761
query36 960 989 909 909
query37 121 105 80 80
query38 4319 4269 4179 4179
query39 1474 1424 1435 1424
query40 211 131 108 108
query41 56 55 57 55
query42 130 117 117 117
query43 524 513 498 498
query44 1382 838 846 838
query45 183 186 179 179
query46 828 1026 654 654
query47 1828 1863 1824 1824
query48 386 424 314 314
query49 692 501 451 451
query50 655 727 398 398
query51 4233 4255 4372 4255
query52 113 113 98 98
query53 243 253 182 182
query54 579 599 517 517
query55 84 84 83 83
query56 316 303 292 292
query57 1161 1196 1162 1162
query58 276 291 261 261
query59 2730 3005 2746 2746
query60 352 326 338 326
query61 136 133 124 124
query62 743 716 692 692
query63 233 185 188 185
query64 1706 1016 669 669
query65 4356 4259 4248 4248
query66 819 395 322 322
query67 15661 15490 15219 15219
query68 7918 872 511 511
query69 586 306 269 269
query70 1213 1143 1097 1097
query71 497 314 307 307
query72 5456 4779 4952 4779
query73 1306 694 360 360
query74 9143 9198 8690 8690
query75 3675 3177 2685 2685
query76 4292 1081 649 649
query77 623 368 288 288
query78 8995 8883 8327 8327
query79 2059 877 564 564
query80 597 519 458 458
query81 485 250 219 219
query82 465 130 97 97
query83 268 258 238 238
query84 295 112 84 84
query85 870 324 274 274
query86 427 301 303 301
query87 4485 4591 4406 4406
query88 3664 2255 2246 2246
query89 403 316 287 287
query90 1839 217 218 217
query91 144 147 116 116
query92 76 64 59 59
query93 1757 995 572 572
query94 677 397 314 314
query95 379 299 284 284
query96 494 567 283 283
query97 3172 3206 3098 3098
query98 265 200 203 200
query99 1416 1402 1279 1279
Total cold run time: 297229 ms
Total hot run time: 191908 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2796097520
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2872442174
<details>
<summary>TPC-H: <b>Total hot run time: 33841 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea6e2dde86b7743e3f5550e1f93c4874995b9b05, data reload: false
------ Round 1 ----------------------------------
q1 26079 5122 5048 5048
q2 2067 280 188 188
q3 10412 1262 720 720
q4 10233 1008 518 518
q5 7654 2356 2325 2325
q6 183 164 137 137
q7 906 776 615 615
q8 9328 1215 1057 1057
q9 6689 5073 5073 5073
q10 6827 2326 1899 1899
q11 482 294 262 262
q12 347 360 210 210
q13 17774 3669 3096 3096
q14 233 231 220 220
q15 543 493 495 493
q16 434 436 377 377
q17 608 864 387 387
q18 7544 7279 7050 7050
q19 1510 946 539 539
q20 341 334 225 225
q21 4411 3295 2451 2451
q22 1023 1012 951 951
Total cold run time: 115628 ms
Total hot run time: 33841 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5123 5083 5128 5083
q2 235 327 233 233
q3 2156 2669 2294 2294
q4 1332 1768 1365 1365
q5 4541 4498 4357 4357
q6 214 169 125 125
q7 1986 1923 1781 1781
q8 2625 2546 2490 2490
q9 7161 7081 7239 7081
q10 2958 3182 2729 2729
q11 581 511 491 491
q12 688 776 611 611
q13 3464 3891 3289 3289
q14 302 295 277 277
q15 528 471 473 471
q16 442 496 438 438
q17 1172 1560 1342 1342
q18 7734 7384 7553 7384
q19 805 838 918 838
q20 1990 2063 1890 1890
q21 4972 4525 4495 4495
q22 1051 1022 979 979
Total cold run time: 52060 ms
Total hot run time: 50043 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "feiniaofeiafei (via GitHub)" <gi...@apache.org>.
feiniaofeiafei commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2085871023
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -206,7 +213,53 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
+ ExpressionRewriteContext context = new ExpressionRewriteContext(tempCascadeContext);
+ for (int i = 0; i < getOutputs().size(); i++) {
+ Optional<Literal> value = Optional.empty();
+ if (!constantExprsList.isEmpty()) {
+ value = ExpressionUtils.checkConstantExpr(constantExprsList.get(0).get(i), context);
+ if (!value.isPresent()) {
+ continue;
+ }
+ final int fi = i;
+ Literal literal = value.get();
+ if (constantExprsList.stream()
+ .map(exprs -> ExpressionUtils.checkConstantExpr(exprs.get(fi), context))
+ .anyMatch(val -> !val.isPresent() || !val.get().equals(literal))) {
+ continue;
+ }
+ }
+ for (int childIdx = 0; childIdx < children.size(); childIdx++) {
+ // TODO: use originOutputs = child(childIdx).getOutput() ?
+ List<? extends Slot> originOutputs = regularChildrenOutputs.get(childIdx);
+ Slot slot = originOutputs.get(i);
+ Optional<Expression> childValue = child(childIdx).getLogicalProperties()
+ .getTrait().getUniformValue(slot);
+ if (childValue == null || !childValue.isPresent() || !childValue.get().isConstant()) {
+ value = Optional.empty();
+ break;
+ }
+ Expression childExpr = childValue.get();
+ if (!childExpr.isLiteral()) {
+ childExpr = FoldConstantRule.evaluate(childExpr, context);
+ if (!childExpr.isLiteral()) {
+ continue;
Review Comment:
this continue may lead to wrong result, should use value=Optional.empty(); break;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2893846910
<details>
<summary>TPC-H: <b>Total hot run time: 34213 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 077b5b29596ba046f9e36350c68df03dd6caf0aa, data reload: false
------ Round 1 ----------------------------------
q1 26997 5108 5037 5037
q2 2078 290 192 192
q3 10376 1263 709 709
q4 10230 1010 533 533
q5 7531 2479 2372 2372
q6 186 166 133 133
q7 922 758 636 636
q8 9348 1316 1059 1059
q9 6818 5136 5142 5136
q10 6819 2318 1922 1922
q11 481 291 289 289
q12 351 356 213 213
q13 17809 3779 3134 3134
q14 240 235 214 214
q15 552 509 511 509
q16 424 435 379 379
q17 613 873 369 369
q18 7720 7315 7222 7222
q19 1257 963 587 587
q20 363 336 231 231
q21 3851 3215 2357 2357
q22 1059 1020 980 980
Total cold run time: 116025 ms
Total hot run time: 34213 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5124 5092 5166 5092
q2 249 330 234 234
q3 2189 2679 2305 2305
q4 1404 1847 1540 1540
q5 4576 4465 4388 4388
q6 218 172 126 126
q7 1987 1906 1752 1752
q8 2590 2489 2571 2489
q9 7241 7019 7118 7019
q10 3075 3221 2812 2812
q11 573 514 498 498
q12 699 769 613 613
q13 3505 3953 3268 3268
q14 298 293 286 286
q15 554 481 478 478
q16 437 492 439 439
q17 1198 1512 1388 1388
q18 8022 7523 7423 7423
q19 852 818 1013 818
q20 1979 2038 1842 1842
q21 4868 4435 4311 4311
q22 1093 1063 1013 1013
Total cold run time: 52731 ms
Total hot run time: 50134 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2898378365
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2898374558
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2919124312
<details>
<summary>ClickBench: <b>Total hot run time: 29.49 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72063f534c073770ef22ff79f1cb2eb3c505b7e7, data reload: false
query1 0.03 0.03 0.03
query2 0.12 0.11 0.11
query3 0.26 0.18 0.19
query4 1.59 0.19 0.20
query5 0.46 0.45 0.44
query6 1.49 0.66 0.66
query7 0.03 0.01 0.02
query8 0.04 0.04 0.04
query9 0.56 0.52 0.52
query10 0.56 0.57 0.57
query11 0.15 0.11 0.12
query12 0.15 0.12 0.12
query13 0.63 0.61 0.60
query14 0.81 0.81 0.80
query15 0.91 0.84 0.87
query16 0.38 0.39 0.40
query17 1.06 1.02 1.04
query18 0.22 0.21 0.22
query19 2.01 1.83 1.83
query20 0.02 0.01 0.02
query21 15.40 0.90 0.55
query22 0.74 1.21 0.69
query23 14.90 1.37 0.62
query24 6.46 1.94 1.02
query25 0.74 0.06 0.10
query26 0.88 0.17 0.13
query27 0.05 0.05 0.06
query28 10.17 0.92 0.45
query29 12.59 4.00 3.33
query30 0.26 0.09 0.06
query31 2.81 0.60 0.39
query32 3.24 0.56 0.48
query33 3.14 3.12 3.14
query34 15.71 5.10 4.53
query35 4.51 4.50 4.55
query36 0.66 0.49 0.48
query37 0.09 0.07 0.07
query38 0.05 0.04 0.03
query39 0.04 0.03 0.03
query40 0.16 0.14 0.13
query41 0.07 0.03 0.03
query42 0.04 0.03 0.02
query43 0.04 0.04 0.04
Total cold run time: 104.23 s
Total hot run time: 29.49 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2944016954
<details>
<summary>TPC-H: <b>Total hot run time: 33860 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c0c5557ef07d1fbe8f42072b38acc17bfaed12bd, data reload: false
------ Round 1 ----------------------------------
q1 26442 5082 5040 5040
q2 1978 287 186 186
q3 10242 1225 721 721
q4 10245 988 534 534
q5 7598 2384 2344 2344
q6 189 168 134 134
q7 903 761 629 629
q8 9315 1219 1076 1076
q9 6881 5109 5127 5109
q10 6865 2326 1897 1897
q11 480 301 282 282
q12 343 349 210 210
q13 17764 3667 3100 3100
q14 233 245 225 225
q15 577 486 505 486
q16 429 434 371 371
q17 591 841 358 358
q18 7464 7067 7017 7017
q19 1264 964 559 559
q20 346 337 222 222
q21 3952 3201 2390 2390
q22 1081 1025 970 970
Total cold run time: 115182 ms
Total hot run time: 33860 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5138 5068 5037 5037
q2 243 321 226 226
q3 2124 2674 2294 2294
q4 1417 1784 1385 1385
q5 4478 4402 4419 4402
q6 223 174 131 131
q7 2010 1901 1751 1751
q8 2589 2580 2528 2528
q9 7239 7265 6936 6936
q10 3037 3224 2829 2829
q11 576 523 500 500
q12 677 750 611 611
q13 3553 3977 3277 3277
q14 298 292 283 283
q15 537 480 497 480
q16 463 502 434 434
q17 1157 1545 1373 1373
q18 7748 7575 7517 7517
q19 830 851 958 851
q20 2010 2116 1869 1869
q21 4854 4524 4379 4379
q22 1102 1083 1053 1053
Total cold run time: 52303 ms
Total hot run time: 50146 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2954014086
<details>
<summary>TPC-DS: <b>Total hot run time: 191640 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 494426d754a065a8b4eb6c2a4a7d4ee6133193db, data reload: false
query1 1452 1104 1056 1056
query2 6293 1804 1788 1788
query3 11052 4465 4642 4465
query4 54131 25614 23296 23296
query5 5086 527 476 476
query6 350 226 205 205
query7 4900 495 301 301
query8 299 231 221 221
query9 5407 2614 2602 2602
query10 454 331 272 272
query11 15299 15114 14810 14810
query12 158 111 106 106
query13 1031 522 430 430
query14 10326 6272 6320 6272
query15 200 217 194 194
query16 7153 656 493 493
query17 1072 756 579 579
query18 1592 414 305 305
query19 193 187 172 172
query20 129 125 113 113
query21 210 126 107 107
query22 4589 4598 4442 4442
query23 34346 33650 33583 33583
query24 6514 2419 2363 2363
query25 495 465 412 412
query26 672 297 149 149
query27 2065 501 351 351
query28 2968 2162 2145 2145
query29 581 606 449 449
query30 274 226 191 191
query31 875 816 737 737
query32 72 71 64 64
query33 467 360 333 333
query34 783 883 541 541
query35 821 856 757 757
query36 966 997 900 900
query37 119 103 77 77
query38 4223 4321 4298 4298
query39 1516 1513 1445 1445
query40 209 121 112 112
query41 63 63 65 63
query42 122 108 107 107
query43 511 503 481 481
query44 1351 852 854 852
query45 187 186 184 184
query46 864 1025 667 667
query47 1871 1863 1796 1796
query48 414 447 330 330
query49 664 504 414 414
query50 649 701 423 423
query51 4203 4242 4177 4177
query52 112 113 108 108
query53 224 273 193 193
query54 593 585 509 509
query55 91 98 96 96
query56 351 299 313 299
query57 1225 1205 1126 1126
query58 276 268 260 260
query59 2674 2761 2686 2686
query60 332 338 315 315
query61 141 144 147 144
query62 733 759 680 680
query63 230 197 198 197
query64 1513 1039 669 669
query65 4198 4137 4162 4137
query66 721 390 301 301
query67 16004 15872 15441 15441
query68 7552 888 531 531
query69 566 308 277 277
query70 1174 1121 1093 1093
query71 500 322 295 295
query72 6183 4833 4763 4763
query73 1519 636 351 351
query74 9001 9068 8644 8644
query75 3778 3219 2687 2687
query76 4169 1208 729 729
query77 613 378 283 283
query78 9424 9664 8639 8639
query79 2031 835 564 564
query80 615 513 437 437
query81 475 255 216 216
query82 417 131 103 103
query83 249 252 236 236
query84 298 111 104 104
query85 806 349 308 308
query86 395 303 282 282
query87 4288 4471 4323 4323
query88 3603 2291 2264 2264
query89 410 314 280 280
query90 1936 203 203 203
query91 148 152 118 118
query92 71 64 60 60
query93 1106 931 592 592
query94 684 398 291 291
query95 368 292 294 292
query96 486 567 276 276
query97 2711 2760 2641 2641
query98 248 220 207 207
query99 1651 1394 1284 1284
Total cold run time: 297134 ms
Total hot run time: 191640 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2976161112
<details>
<summary>TPC-H: <b>Total hot run time: 35169 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92f51c7b47bfdc21b0d37aaab60c326abc3681d8, data reload: false
------ Round 1 ----------------------------------
q1 17601 5143 4999 4999
q2 1961 281 176 176
q3 10319 1266 769 769
q4 10233 1039 529 529
q5 7747 2380 2370 2370
q6 186 165 137 137
q7 924 736 605 605
q8 9336 1326 1185 1185
q9 6769 5095 5091 5091
q10 6903 2327 1891 1891
q11 503 308 290 290
q12 355 364 242 242
q13 17786 3686 3139 3139
q14 243 231 220 220
q15 578 479 482 479
q16 435 453 399 399
q17 683 873 416 416
q18 7568 7449 7233 7233
q19 2056 988 578 578
q20 367 354 246 246
q21 4290 3540 3181 3181
q22 1101 1050 994 994
Total cold run time: 107944 ms
Total hot run time: 35169 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5093 5121 5014 5014
q2 255 344 227 227
q3 2274 2746 2362 2362
q4 1433 1840 1430 1430
q5 4315 4376 4401 4376
q6 223 170 137 137
q7 2028 1934 1774 1774
q8 2654 2607 2598 2598
q9 7229 7249 7253 7249
q10 2965 3260 2794 2794
q11 602 531 520 520
q12 687 784 645 645
q13 3584 3911 3301 3301
q14 280 319 296 296
q15 549 498 467 467
q16 470 491 458 458
q17 1197 1605 1373 1373
q18 7848 7487 7628 7487
q19 916 913 971 913
q20 2006 2128 1933 1933
q21 4838 4613 4462 4462
q22 1132 1064 1051 1051
Total cold run time: 52578 ms
Total hot run time: 50867 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2798469967
<details>
<summary>ClickBench: <b>Total hot run time: 30.98 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 679002894d5fb311f725de08d1ffda48e95234fc, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.10 0.12
query3 0.26 0.20 0.19
query4 1.59 0.20 0.19
query5 0.61 0.60 0.58
query6 1.18 0.73 0.72
query7 0.02 0.02 0.02
query8 0.04 0.04 0.03
query9 0.58 0.51 0.51
query10 0.58 0.57 0.59
query11 0.15 0.10 0.11
query12 0.15 0.12 0.12
query13 0.61 0.60 0.61
query14 2.72 2.72 2.70
query15 0.94 0.86 0.85
query16 0.40 0.40 0.38
query17 1.05 1.04 1.00
query18 0.21 0.19 0.20
query19 1.84 1.92 1.86
query20 0.02 0.01 0.02
query21 15.36 0.88 0.55
query22 0.76 1.08 0.60
query23 15.12 1.43 0.64
query24 7.19 1.28 0.58
query25 0.46 0.20 0.07
query26 0.61 0.17 0.15
query27 0.05 0.05 0.05
query28 8.66 0.90 0.45
query29 12.58 3.98 3.30
query30 0.25 0.09 0.06
query31 2.81 0.58 0.39
query32 3.24 0.56 0.47
query33 3.05 3.20 3.14
query34 15.78 5.08 4.47
query35 4.53 4.55 4.54
query36 0.65 0.50 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.03
query39 0.03 0.03 0.02
query40 0.17 0.14 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.03
query43 0.04 0.03 0.02
Total cold run time: 104.71 s
Total hot run time: 30.98 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838967830
<details>
<summary>ClickBench: <b>Total hot run time: 29.17 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ab930e4985c14df6decf4c7d449aea9ef0685c2, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.12
query3 0.25 0.19 0.19
query4 1.59 0.21 0.11
query5 0.56 0.54 0.55
query6 1.20 0.71 0.72
query7 0.02 0.01 0.01
query8 0.04 0.04 0.03
query9 0.56 0.51 0.50
query10 0.58 0.57 0.57
query11 0.15 0.11 0.11
query12 0.15 0.11 0.11
query13 0.62 0.59 0.59
query14 0.79 0.79 0.80
query15 0.88 0.85 0.86
query16 0.38 0.39 0.39
query17 1.04 1.04 1.01
query18 0.21 0.20 0.21
query19 1.86 1.84 1.78
query20 0.02 0.01 0.01
query21 15.40 0.96 0.55
query22 0.74 1.22 0.69
query23 14.91 1.37 0.64
query24 7.04 1.66 0.58
query25 0.51 0.22 0.10
query26 0.60 0.17 0.14
query27 0.05 0.06 0.05
query28 9.90 0.83 0.45
query29 12.57 4.19 3.46
query30 0.28 0.09 0.07
query31 2.83 0.61 0.39
query32 3.23 0.54 0.45
query33 3.01 3.02 3.12
query34 15.84 5.15 4.56
query35 4.53 4.59 4.60
query36 0.67 0.49 0.48
query37 0.09 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.02 0.02
query40 0.16 0.14 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.03
query43 0.04 0.03 0.03
Total cold run time: 103.66 s
Total hot run time: 29.17 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2824723458
<details>
<summary>TPC-DS: <b>Total hot run time: 185039 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12489c4b0c61b9e294caec8c15a297f2c7171fbc, data reload: false
query1 1003 489 490 489
query2 6552 1765 1737 1737
query3 6737 234 218 218
query4 26247 23786 22935 22935
query5 4392 649 468 468
query6 286 202 202 202
query7 4624 498 277 277
query8 281 241 242 241
query9 8624 2546 2554 2546
query10 488 340 279 279
query11 15203 15117 14849 14849
query12 164 111 107 107
query13 1664 535 398 398
query14 8694 6234 6300 6234
query15 213 197 173 173
query16 7321 665 497 497
query17 1210 728 586 586
query18 1969 409 304 304
query19 192 188 162 162
query20 122 120 118 118
query21 213 133 111 111
query22 4132 4249 4012 4012
query23 33948 32965 32895 32895
query24 8283 2397 2432 2397
query25 533 463 396 396
query26 1243 270 152 152
query27 2734 497 345 345
query28 4315 2104 2087 2087
query29 764 546 448 448
query30 274 254 181 181
query31 904 801 732 732
query32 77 66 66 66
query33 549 354 319 319
query34 785 838 521 521
query35 780 797 727 727
query36 959 1001 907 907
query37 110 101 79 79
query38 4251 4172 4208 4172
query39 1438 1402 1415 1402
query40 209 126 102 102
query41 58 56 54 54
query42 120 110 105 105
query43 493 499 449 449
query44 1314 806 819 806
query45 183 174 169 169
query46 839 1027 637 637
query47 1734 1808 1713 1713
query48 374 416 308 308
query49 795 505 434 434
query50 658 701 395 395
query51 4122 4087 4010 4010
query52 108 110 107 107
query53 229 258 191 191
query54 576 577 514 514
query55 88 77 81 77
query56 307 310 320 310
query57 1108 1147 1095 1095
query58 268 258 270 258
query59 2594 2621 2460 2460
query60 312 312 301 301
query61 137 132 124 124
query62 804 742 679 679
query63 227 194 186 186
query64 4359 1034 672 672
query65 4327 4251 4244 4244
query66 1153 411 305 305
query67 15587 15423 15239 15239
query68 8533 882 517 517
query69 462 296 277 277
query70 1231 1165 1054 1054
query71 483 335 307 307
query72 5528 4794 4870 4794
query73 751 667 405 405
query74 8853 9100 8968 8968
query75 3895 3220 2711 2711
query76 3627 1205 759 759
query77 827 379 292 292
query78 9314 9450 8645 8645
query79 2365 837 577 577
query80 575 565 449 449
query81 457 254 226 226
query82 470 129 98 98
query83 286 247 239 239
query84 294 104 92 92
query85 751 325 289 289
query86 345 297 291 291
query87 4450 4452 4367 4367
query88 3086 2233 2237 2233
query89 399 310 282 282
query90 1932 210 213 210
query91 147 146 111 111
query92 76 64 58 58
query93 1337 974 583 583
query94 671 420 319 319
query95 380 294 278 278
query96 486 563 278 278
query97 3144 3232 3143 3143
query98 238 212 204 204
query99 1438 1384 1270 1270
Total cold run time: 272192 ms
Total hot run time: 185039 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209105213
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
+ List<Expression> conjunctions,
+ boolean useInnerInfer,
+ ImmutableEqualSet<Slot> parentEqualSet,
+ Map<Slot, Literal> parentConstants) {
+ Map<Slot, Literal> newConstants = Maps.newLinkedHashMapWithExpectedSize(parentConstants.size());
+ newConstants.putAll(parentConstants);
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>(parentEqualSet);
+ for (Expression child : conjunctions) {
+ Optional<Pair<Slot, Expression>> equalItem = findValidEqualItem(child);
+ if (!equalItem.isPresent()) {
+ continue;
+ }
+ Slot slot = equalItem.get().first;
+ Expression expr = equalItem.get().second;
+ // for expression `a = 1 and a * 3 = 10`
+ // if it's in LogicalFilter, then we can infer constant relation `a = 1`, then have:
+ // `a = 1 and 3 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
Review Comment:
fix
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
+ List<Expression> conjunctions,
+ boolean useInnerInfer,
+ ImmutableEqualSet<Slot> parentEqualSet,
+ Map<Slot, Literal> parentConstants) {
+ Map<Slot, Literal> newConstants = Maps.newLinkedHashMapWithExpectedSize(parentConstants.size());
+ newConstants.putAll(parentConstants);
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>(parentEqualSet);
+ for (Expression child : conjunctions) {
+ Optional<Pair<Slot, Expression>> equalItem = findValidEqualItem(child);
+ if (!equalItem.isPresent()) {
+ continue;
+ }
+ Slot slot = equalItem.get().first;
+ Expression expr = equalItem.get().second;
+ // for expression `a = 1 and a * 3 = 10`
+ // if it's in LogicalFilter, then we can infer constant relation `a = 1`, then have:
+ // `a = 1 and 3 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
+ // but if it's in LogicalProject, then we shouldn't propagate `a=1` to this expression,
+ // because a may be null, then `a=1 and a * 3 = 10` should evaluate to `NULL` in the project.
+ if (!useInnerInfer && (slot.nullable() || expr.nullable())) {
+ continue;
+ }
+ if (expr instanceof Slot) {
+ newEqualSetBuilder.addEqualPair(slot, (Slot) expr);
+ } else if (!addConstant(newConstants, slot, (Literal) expr)) {
+ return Optional.empty();
+ }
+ }
+
+ ImmutableEqualSet<Slot> newEqualSet = newEqualSetBuilder.build();
+ List<Set<Slot>> multiEqualSlots = newEqualSet.calEqualSetList();
+ for (Set<Slot> slots : multiEqualSlots) {
+ Slot slot = slots.stream().filter(newConstants::containsKey).findFirst().orElse(null);
+ if (slot == null) {
+ continue;
+ }
+ Literal value = newConstants.get(slot);
+ for (Slot s : slots) {
+ if (!addConstant(newConstants, s, value)) {
+ return Optional.empty();
+ }
+ }
+ }
+
+ return Optional.of(Pair.of(newEqualSet, newConstants));
+ }
+
+ private boolean addConstant(Map<Slot, Literal> constants, Slot slot, Literal value) {
Review Comment:
had add annotation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076629186
<details>
<summary>ClickBench: <b>Total hot run time: 32.08 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79a5db32079ea3bcb6f30e554e2924fe379da810, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.04
query3 0.25 0.07 0.07
query4 1.62 0.11 0.11
query5 0.45 0.42 0.42
query6 1.20 0.67 0.66
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.61 0.52 0.51
query10 0.57 0.60 0.57
query11 0.16 0.11 0.11
query12 0.15 0.13 0.11
query13 0.62 0.62 0.62
query14 0.82 0.82 0.80
query15 0.90 0.86 0.89
query16 0.38 0.39 0.38
query17 1.08 1.05 1.06
query18 0.23 0.22 0.21
query19 1.93 1.81 1.89
query20 0.01 0.02 0.01
query21 15.39 0.88 0.52
query22 0.75 1.28 0.66
query23 14.88 1.40 0.62
query24 8.84 0.62 0.42
query25 0.43 0.18 0.21
query26 0.75 0.16 0.13
query27 0.07 0.05 0.06
query28 8.81 0.85 0.46
query29 12.53 4.00 3.31
query30 3.07 3.00 2.93
query31 2.82 0.56 0.38
query32 3.28 0.55 0.48
query33 3.18 3.09 3.16
query34 15.99 5.36 4.79
query35 4.84 4.80 4.78
query36 0.68 0.52 0.48
query37 0.10 0.07 0.07
query38 0.06 0.05 0.05
query39 0.03 0.03 0.02
query40 0.17 0.15 0.15
query41 0.08 0.03 0.03
query42 0.04 0.03 0.02
query43 0.04 0.03 0.04
Total cold run time: 108.01 s
Total hot run time: 32.08 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076490105
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3079311601
<details>
<summary>ClickBench: <b>Total hot run time: 32.51 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a20e730c75a1273b1167641933d055aea6470f95, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.04 0.05
query3 0.25 0.07 0.07
query4 1.62 0.11 0.12
query5 0.45 0.42 0.41
query6 1.16 0.65 0.68
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.59 0.51 0.50
query10 0.59 0.56 0.58
query11 0.17 0.11 0.11
query12 0.15 0.12 0.12
query13 0.62 0.60 0.62
query14 0.79 0.82 0.82
query15 0.90 0.87 0.86
query16 0.40 0.38 0.40
query17 1.04 1.04 1.07
query18 0.23 0.22 0.22
query19 1.94 1.82 1.83
query20 0.01 0.01 0.02
query21 15.38 0.89 0.55
query22 0.75 1.17 0.66
query23 14.96 1.37 0.60
query24 6.62 2.18 0.86
query25 0.48 0.31 0.15
query26 0.60 0.16 0.14
query27 0.06 0.05 0.05
query28 9.35 0.86 0.46
query29 12.55 3.98 3.27
query30 3.07 2.98 2.97
query31 2.82 0.58 0.39
query32 3.24 0.56 0.47
query33 3.05 3.08 3.11
query34 16.03 5.33 4.79
query35 4.78 4.83 4.83
query36 0.70 0.50 0.50
query37 0.09 0.07 0.08
query38 0.05 0.05 0.03
query39 0.04 0.03 0.02
query40 0.17 0.15 0.13
query41 0.08 0.03 0.03
query42 0.03 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 106.03 s
Total hot run time: 32.51 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3100790306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2830004418
<details>
<summary>ClickBench: <b>Total hot run time: 29.26 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7c93fc833a1a8da818b939c19fbf70373fb15231, data reload: false
query1 0.03 0.04 0.02
query2 0.12 0.11 0.12
query3 0.26 0.20 0.20
query4 1.59 0.19 0.20
query5 0.59 0.59 0.60
query6 1.20 0.72 0.72
query7 0.02 0.02 0.02
query8 0.04 0.04 0.04
query9 0.58 0.52 0.51
query10 0.57 0.57 0.57
query11 0.15 0.11 0.11
query12 0.15 0.11 0.11
query13 0.61 0.60 0.60
query14 1.19 1.15 1.16
query15 0.87 0.85 0.88
query16 0.39 0.39 0.41
query17 1.03 1.04 1.05
query18 0.21 0.20 0.19
query19 1.92 1.78 1.78
query20 0.01 0.02 0.01
query21 15.44 0.86 0.54
query22 0.78 1.11 0.69
query23 14.92 1.38 0.61
query24 6.62 1.95 0.48
query25 0.55 0.17 0.06
query26 0.52 0.17 0.14
query27 0.05 0.06 0.05
query28 9.42 0.91 0.42
query29 12.58 4.02 3.29
query30 0.25 0.09 0.07
query31 2.81 0.61 0.40
query32 3.24 0.55 0.46
query33 3.05 2.98 3.12
query34 15.82 5.14 4.54
query35 4.56 4.55 4.54
query36 0.66 0.50 0.47
query37 0.09 0.07 0.06
query38 0.04 0.04 0.04
query39 0.03 0.02 0.03
query40 0.17 0.14 0.14
query41 0.08 0.02 0.02
query42 0.04 0.02 0.03
query43 0.04 0.04 0.02
Total cold run time: 103.29 s
Total hot run time: 29.26 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2823061556
<details>
<summary>ClickBench: <b>Total hot run time: 29.79 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb7fcae3e6afa2a5aa88d0278ff4850df770d248, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.11
query3 0.25 0.20 0.19
query4 1.60 0.18 0.20
query5 0.57 0.58 0.59
query6 1.21 0.73 0.73
query7 0.03 0.02 0.01
query8 0.05 0.04 0.04
query9 0.55 0.52 0.51
query10 0.57 0.56 0.58
query11 0.15 0.12 0.11
query12 0.14 0.11 0.11
query13 0.61 0.60 0.61
query14 1.17 1.22 1.22
query15 0.88 0.86 0.86
query16 0.40 0.39 0.38
query17 1.03 1.04 1.02
query18 0.21 0.20 0.19
query19 1.90 1.72 1.78
query20 0.02 0.01 0.02
query21 15.41 0.95 0.55
query22 0.76 1.23 0.67
query23 14.88 1.40 0.65
query24 7.14 1.24 0.92
query25 0.47 0.13 0.25
query26 0.59 0.16 0.13
query27 0.05 0.05 0.05
query28 9.57 0.91 0.44
query29 12.59 4.01 3.28
query30 0.25 0.09 0.06
query31 2.84 0.58 0.38
query32 3.23 0.56 0.48
query33 3.06 3.10 3.11
query34 15.73 5.12 4.51
query35 4.48 4.54 4.50
query36 0.66 0.50 0.47
query37 0.09 0.06 0.07
query38 0.04 0.04 0.03
query39 0.03 0.03 0.03
query40 0.17 0.13 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 103.69 s
Total hot run time: 29.79 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2826910544
<details>
<summary>TPC-DS: <b>Total hot run time: 185347 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 567137845144acf42624ab070da50afc03607d57, data reload: false
query1 1025 484 493 484
query2 6567 1816 1798 1798
query3 6747 222 226 222
query4 26457 23604 23325 23325
query5 4379 629 457 457
query6 302 208 187 187
query7 4623 488 277 277
query8 286 236 241 236
query9 8616 2583 2607 2583
query10 483 347 270 270
query11 15428 15052 15293 15052
query12 170 112 114 112
query13 1655 522 399 399
query14 9153 6218 6127 6127
query15 197 187 178 178
query16 7173 643 485 485
query17 1227 764 574 574
query18 1986 408 296 296
query19 189 191 165 165
query20 118 128 124 124
query21 219 129 106 106
query22 4152 4175 4106 4106
query23 33978 33020 33012 33012
query24 8477 2456 2380 2380
query25 534 445 390 390
query26 1242 269 154 154
query27 2754 503 325 325
query28 4339 2114 2101 2101
query29 755 551 429 429
query30 279 213 193 193
query31 956 846 728 728
query32 74 68 62 62
query33 555 357 318 318
query34 793 833 532 532
query35 774 826 719 719
query36 963 1013 910 910
query37 117 98 72 72
query38 4114 4212 4143 4143
query39 1463 1376 1391 1376
query40 213 119 106 106
query41 60 58 53 53
query42 120 112 109 109
query43 496 486 465 465
query44 1287 824 812 812
query45 180 176 174 174
query46 839 1011 622 622
query47 1771 1849 1756 1756
query48 364 417 314 314
query49 782 504 444 444
query50 630 669 399 399
query51 4076 4116 4036 4036
query52 110 110 96 96
query53 222 255 184 184
query54 576 558 505 505
query55 85 82 85 82
query56 323 290 284 284
query57 1118 1166 1081 1081
query58 270 256 257 256
query59 2587 2733 2580 2580
query60 348 317 304 304
query61 133 126 128 126
query62 779 716 667 667
query63 217 188 190 188
query64 4344 1009 682 682
query65 4310 4188 4258 4188
query66 1164 408 301 301
query67 15814 15372 15336 15336
query68 8350 889 511 511
query69 519 302 260 260
query70 1220 1131 1111 1111
query71 465 313 304 304
query72 5313 4761 4623 4623
query73 713 562 351 351
query74 8879 8820 8940 8820
query75 3943 3187 2724 2724
query76 3731 1208 737 737
query77 792 385 286 286
query78 9284 9504 8650 8650
query79 1597 866 556 556
query80 585 544 439 439
query81 473 257 216 216
query82 420 129 96 96
query83 260 253 246 246
query84 256 172 84 84
query85 745 328 281 281
query86 337 293 296 293
query87 4321 4428 4320 4320
query88 2832 2237 2198 2198
query89 382 321 271 271
query90 1930 206 208 206
query91 141 145 111 111
query92 75 61 62 61
query93 1038 949 573 573
query94 661 414 299 299
query95 371 305 284 284
query96 482 563 273 273
query97 3162 3188 3118 3118
query98 221 214 209 209
query99 1445 1385 1279 1279
Total cold run time: 271480 ms
Total hot run time: 185347 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833426289
<details>
<summary>TPC-DS: <b>Total hot run time: 191838 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 623b77512221f6c2e11c5e755e77011eb36e2a01, data reload: false
query1 1409 1104 1086 1086
query2 6219 1852 1870 1852
query3 11187 4641 4601 4601
query4 25934 23756 23265 23265
query5 5537 617 455 455
query6 295 197 215 197
query7 3993 500 293 293
query8 313 261 238 238
query9 8830 2569 2602 2569
query10 533 319 266 266
query11 15298 15126 14773 14773
query12 163 115 104 104
query13 1573 545 408 408
query14 9195 6233 6097 6097
query15 201 188 175 175
query16 7585 659 484 484
query17 1091 741 592 592
query18 2100 435 304 304
query19 193 185 176 176
query20 123 125 116 116
query21 207 131 110 110
query22 4415 4634 4152 4152
query23 34658 33670 33642 33642
query24 8532 2454 2405 2405
query25 507 454 396 396
query26 1214 276 157 157
query27 2973 505 339 339
query28 4557 2168 2150 2150
query29 749 591 449 449
query30 285 221 187 187
query31 896 839 756 756
query32 72 66 67 66
query33 563 371 332 332
query34 797 885 539 539
query35 786 846 751 751
query36 971 1025 897 897
query37 111 106 79 79
query38 4257 4321 4235 4235
query39 1472 1499 1432 1432
query40 217 120 114 114
query41 57 57 53 53
query42 118 106 103 103
query43 519 517 495 495
query44 1335 845 850 845
query45 190 180 183 180
query46 843 1064 648 648
query47 1896 1860 1804 1804
query48 383 421 329 329
query49 776 538 443 443
query50 657 697 448 448
query51 4227 4252 4134 4134
query52 112 108 98 98
query53 226 266 185 185
query54 606 574 507 507
query55 87 98 80 80
query56 300 302 301 301
query57 1204 1193 1137 1137
query58 261 253 263 253
query59 2905 2842 2845 2842
query60 365 338 308 308
query61 152 152 172 152
query62 726 754 655 655
query63 220 176 185 176
query64 3977 1043 654 654
query65 4350 4277 4274 4274
query66 899 389 297 297
query67 15657 15412 15292 15292
query68 8437 877 514 514
query69 429 311 263 263
query70 1209 1124 1089 1089
query71 409 321 305 305
query72 5710 4791 4874 4791
query73 685 647 348 348
query74 8891 9323 8692 8692
query75 3179 3170 2709 2709
query76 2959 1174 801 801
query77 562 360 289 289
query78 9586 9529 8669 8669
query79 1834 874 563 563
query80 689 512 524 512
query81 503 254 223 223
query82 407 125 98 98
query83 254 247 240 240
query84 253 100 90 90
query85 722 331 282 282
query86 339 287 284 284
query87 4304 4400 4254 4254
query88 2800 2174 2178 2174
query89 384 339 280 280
query90 1770 205 206 205
query91 155 140 112 112
query92 60 60 57 57
query93 1299 983 579 579
query94 629 406 312 312
query95 368 287 280 280
query96 480 547 270 270
query97 3116 3241 3131 3131
query98 221 211 225 211
query99 1350 1418 1279 1279
Total cold run time: 277052 ms
Total hot run time: 191838 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3043599590
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2804507698
<details>
<summary>TPC-H: <b>Total hot run time: 33924 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2930d988d2c992f3fe1a6254e06d831d20a0a171, data reload: false
------ Round 1 ----------------------------------
q1 25910 5004 4991 4991
q2 2083 290 185 185
q3 10377 1206 684 684
q4 10222 1019 564 564
q5 7539 2319 2323 2319
q6 186 162 134 134
q7 912 714 607 607
q8 9300 1236 1051 1051
q9 6777 5082 5084 5082
q10 6801 2298 1896 1896
q11 485 290 267 267
q12 349 345 212 212
q13 17778 3698 3105 3105
q14 237 228 213 213
q15 527 478 493 478
q16 625 611 572 572
q17 572 843 351 351
q18 7435 7079 7135 7079
q19 1251 947 549 549
q20 336 322 208 208
q21 3888 3366 2416 2416
q22 1071 1040 961 961
Total cold run time: 114661 ms
Total hot run time: 33924 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5068 5379 5115 5115
q2 237 326 232 232
q3 2271 2702 2285 2285
q4 1498 1844 1457 1457
q5 4501 4465 4443 4443
q6 214 170 130 130
q7 2075 1979 1814 1814
q8 2647 2569 2588 2569
q9 7430 7389 7256 7256
q10 3001 3169 2739 2739
q11 576 522 494 494
q12 701 762 628 628
q13 3463 3876 3335 3335
q14 301 307 296 296
q15 522 497 477 477
q16 656 702 646 646
q17 1147 1513 1409 1409
q18 7759 7512 7414 7414
q19 795 775 779 775
q20 1925 1935 1810 1810
q21 5321 4930 4772 4772
q22 1125 1049 1059 1049
Total cold run time: 53233 ms
Total hot run time: 51145 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2832972939
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2954018462
<details>
<summary>ClickBench: <b>Total hot run time: 29.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 494426d754a065a8b4eb6c2a4a7d4ee6133193db, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.12
query3 0.25 0.20 0.20
query4 1.60 0.19 0.20
query5 0.48 0.43 0.44
query6 1.18 0.66 0.66
query7 0.02 0.01 0.01
query8 0.04 0.04 0.04
query9 0.59 0.51 0.51
query10 0.58 0.56 0.58
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.62 0.60 0.60
query14 0.79 0.81 0.80
query15 0.88 0.87 0.86
query16 0.38 0.40 0.38
query17 1.07 1.09 1.06
query18 0.23 0.21 0.21
query19 1.93 1.79 1.78
query20 0.01 0.01 0.01
query21 15.40 0.93 0.53
query22 0.76 1.15 0.60
query23 15.02 1.40 0.62
query24 6.79 1.97 0.88
query25 0.49 0.23 0.07
query26 0.50 0.17 0.14
query27 0.05 0.06 0.05
query28 9.89 0.90 0.45
query29 12.59 4.07 3.48
query30 0.26 0.09 0.06
query31 2.82 0.62 0.39
query32 3.23 0.55 0.47
query33 3.14 3.06 3.15
query34 15.71 5.12 4.47
query35 4.55 4.49 4.52
query36 0.66 0.50 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.16 0.14 0.13
query41 0.07 0.02 0.02
query42 0.04 0.03 0.03
query43 0.04 0.03 0.03
Total cold run time: 103.44 s
Total hot run time: 29.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2976046619
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3086541516
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3086623876
<details>
<summary>TPC-H: <b>Total hot run time: 34188 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb92819243a981911b585dcecfc6e8655c8959c7, data reload: false
------ Round 1 ----------------------------------
q1 17605 5191 5058 5058
q2 1945 301 195 195
q3 10282 1296 718 718
q4 10225 1034 546 546
q5 7528 2376 2325 2325
q6 190 194 138 138
q7 922 762 628 628
q8 9340 1312 1082 1082
q9 7026 5138 5144 5138
q10 6901 2411 1967 1967
q11 487 308 288 288
q12 367 376 234 234
q13 17771 3671 3093 3093
q14 254 238 237 237
q15 580 528 535 528
q16 425 451 377 377
q17 589 862 357 357
q18 7772 7252 7205 7205
q19 1102 972 565 565
q20 334 365 227 227
q21 3860 3158 2322 2322
q22 1030 1030 960 960
Total cold run time: 106535 ms
Total hot run time: 34188 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5159 5096 5143 5096
q2 259 340 233 233
q3 2200 2640 2267 2267
q4 1309 1770 1271 1271
q5 4216 4558 4548 4548
q6 222 184 138 138
q7 2011 1914 1828 1828
q8 2627 2649 2570 2570
q9 7268 7288 7332 7288
q10 3061 3300 2885 2885
q11 579 522 517 517
q12 713 800 635 635
q13 3510 3901 3307 3307
q14 285 305 282 282
q15 579 512 507 507
q16 458 505 566 505
q17 1304 1566 1362 1362
q18 7808 7630 7675 7630
q19 857 952 971 952
q20 2019 2095 1916 1916
q21 5072 4584 4542 4542
q22 1142 1116 1119 1116
Total cold run time: 52658 ms
Total hot run time: 51395 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076462752
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209102473
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
Review Comment:
udpate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076545727
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209658171
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
Review Comment:
had rewrite mark conditions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3078858534
<details>
<summary>ClickBench: <b>Total hot run time: 31.8 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7dd3d7e2d7f78983d969a79698664d3686e15112, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.05 0.04
query3 0.24 0.07 0.07
query4 1.63 0.11 0.11
query5 0.45 0.42 0.40
query6 1.16 0.65 0.67
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.58 0.50 0.51
query10 0.57 0.58 0.57
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.63 0.60 0.62
query14 0.80 0.81 0.81
query15 0.90 0.85 0.87
query16 0.39 0.39 0.40
query17 1.04 1.05 1.10
query18 0.23 0.22 0.21
query19 2.00 1.84 1.80
query20 0.01 0.00 0.01
query21 15.40 0.91 0.53
query22 0.76 1.08 0.59
query23 15.12 1.37 0.59
query24 7.47 1.06 0.36
query25 0.52 0.30 0.07
query26 0.61 0.16 0.15
query27 0.06 0.05 0.05
query28 9.35 0.89 0.45
query29 12.60 3.92 3.27
query30 3.02 2.97 2.95
query31 2.82 0.59 0.38
query32 3.23 0.55 0.49
query33 3.12 3.12 3.15
query34 16.08 5.38 4.78
query35 4.83 4.89 4.82
query36 0.71 0.52 0.50
query37 0.09 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.02
query40 0.18 0.14 0.14
query41 0.09 0.03 0.03
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 107.33 s
Total hot run time: 31.8 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795751666
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2809373842
<details>
<summary>TPC-H: <b>Total hot run time: 34043 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eaef1200d54db43a8b5ff63aa1975bd8d9b89cfb, data reload: false
------ Round 1 ----------------------------------
q1 26301 5141 5067 5067
q2 2068 281 192 192
q3 10397 1247 676 676
q4 10240 989 519 519
q5 8063 2473 2352 2352
q6 179 163 134 134
q7 930 749 610 610
q8 9311 1282 1151 1151
q9 6841 5155 5185 5155
q10 6897 2352 1896 1896
q11 494 292 265 265
q12 350 357 229 229
q13 17804 3669 3137 3137
q14 228 239 219 219
q15 545 483 479 479
q16 458 450 399 399
q17 579 861 352 352
q18 7535 7265 7056 7056
q19 1362 965 550 550
q20 340 321 224 224
q21 3968 3374 2404 2404
q22 1030 1003 977 977
Total cold run time: 115920 ms
Total hot run time: 34043 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5089 5059 5091 5059
q2 235 327 231 231
q3 2160 2671 2268 2268
q4 1379 1806 1447 1447
q5 4462 4398 4384 4384
q6 217 180 123 123
q7 1986 1923 1783 1783
q8 2588 2635 2547 2547
q9 7274 7259 7013 7013
q10 3041 3159 2730 2730
q11 581 532 488 488
q12 682 782 591 591
q13 3503 3927 3273 3273
q14 278 313 281 281
q15 539 492 472 472
q16 465 495 459 459
q17 1149 1562 1434 1434
q18 7813 7602 7423 7423
q19 786 763 818 763
q20 1982 2046 1843 1843
q21 5173 4930 4797 4797
q22 1133 1051 1000 1000
Total cold run time: 52515 ms
Total hot run time: 50409 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2826924075
<details>
<summary>ClickBench: <b>Total hot run time: 29.64 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 567137845144acf42624ab070da50afc03607d57, data reload: false
query1 0.03 0.04 0.03
query2 0.12 0.10 0.11
query3 0.26 0.19 0.20
query4 1.60 0.19 0.11
query5 0.56 0.54 0.54
query6 1.20 0.72 0.72
query7 0.02 0.01 0.02
query8 0.05 0.03 0.04
query9 0.61 0.54 0.54
query10 0.59 0.60 0.59
query11 0.16 0.12 0.13
query12 0.16 0.11 0.12
query13 0.62 0.59 0.59
query14 1.17 1.18 1.21
query15 0.88 0.85 0.86
query16 0.39 0.38 0.39
query17 1.02 1.03 1.00
query18 0.21 0.20 0.20
query19 1.88 1.83 1.81
query20 0.01 0.02 0.01
query21 15.37 0.89 0.55
query22 0.77 1.17 0.97
query23 14.70 1.38 0.66
query24 7.42 1.17 0.48
query25 0.48 0.27 0.13
query26 0.62 0.17 0.15
query27 0.05 0.06 0.05
query28 9.14 0.93 0.44
query29 12.53 4.05 3.42
query30 0.26 0.08 0.07
query31 2.82 0.61 0.38
query32 3.22 0.54 0.47
query33 2.98 3.02 3.09
query34 15.81 5.10 4.49
query35 4.54 4.46 4.49
query36 0.67 0.51 0.48
query37 0.09 0.07 0.06
query38 0.06 0.03 0.03
query39 0.03 0.03 0.02
query40 0.17 0.14 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.03
query43 0.03 0.03 0.03
Total cold run time: 103.41 s
Total hot run time: 29.64 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2824690584
<details>
<summary>TPC-H: <b>Total hot run time: 34054 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12489c4b0c61b9e294caec8c15a297f2c7171fbc, data reload: false
------ Round 1 ----------------------------------
q1 26174 5076 5034 5034
q2 2074 279 185 185
q3 10398 1254 723 723
q4 10226 1029 534 534
q5 7516 2360 2397 2360
q6 187 167 134 134
q7 943 750 607 607
q8 9333 1301 1117 1117
q9 6807 5174 5112 5112
q10 6839 2301 1879 1879
q11 474 280 262 262
q12 350 352 219 219
q13 17779 3670 3118 3118
q14 224 218 239 218
q15 539 499 484 484
q16 445 447 397 397
q17 575 865 369 369
q18 7476 7335 7041 7041
q19 1217 993 577 577
q20 342 331 227 227
q21 4024 2714 2455 2455
q22 1075 1043 1002 1002
Total cold run time: 115017 ms
Total hot run time: 34054 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5111 5070 5081 5070
q2 247 327 237 237
q3 2155 2675 2270 2270
q4 1465 1833 1400 1400
q5 4521 4528 4409 4409
q6 209 164 129 129
q7 1985 1919 1734 1734
q8 2637 2659 2526 2526
q9 7219 7135 7246 7135
q10 3028 3203 2767 2767
q11 572 514 491 491
q12 668 780 631 631
q13 3480 3910 3243 3243
q14 305 307 270 270
q15 524 497 476 476
q16 469 522 454 454
q17 1148 1551 1386 1386
q18 7757 7541 7497 7497
q19 857 888 963 888
q20 2019 2036 1882 1882
q21 5109 4646 4678 4646
q22 1102 1008 981 981
Total cold run time: 52587 ms
Total hot run time: 50522 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2834932862
<details>
<summary>TPC-DS: <b>Total hot run time: 184516 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 787c34b286f794d7f261bb66d7823155b84c9868, data reload: false
query1 1004 463 488 463
query2 6572 1765 1784 1765
query3 6740 231 226 226
query4 26564 23439 23126 23126
query5 4376 616 459 459
query6 302 228 210 210
query7 4625 487 285 285
query8 297 261 294 261
query9 8635 2583 2589 2583
query10 477 322 272 272
query11 15564 15037 14865 14865
query12 182 107 108 107
query13 1644 533 381 381
query14 9234 5990 6012 5990
query15 188 188 175 175
query16 7176 618 426 426
query17 1182 703 554 554
query18 1979 408 300 300
query19 191 197 165 165
query20 118 116 114 114
query21 210 121 106 106
query22 3968 4156 3910 3910
query23 34032 32948 33183 32948
query24 8436 2355 2344 2344
query25 549 458 416 416
query26 1238 276 157 157
query27 2719 499 335 335
query28 4340 2124 2088 2088
query29 812 533 423 423
query30 288 217 184 184
query31 914 838 707 707
query32 75 68 63 63
query33 558 353 305 305
query34 787 829 523 523
query35 768 809 742 742
query36 959 1052 879 879
query37 109 104 78 78
query38 4161 4158 4166 4158
query39 1469 1461 1396 1396
query40 208 117 112 112
query41 57 57 56 56
query42 116 110 108 108
query43 484 498 476 476
query44 1267 794 786 786
query45 188 173 166 166
query46 830 1023 607 607
query47 1744 1827 1718 1718
query48 373 423 294 294
query49 814 528 430 430
query50 624 680 406 406
query51 4064 4095 4041 4041
query52 107 104 95 95
query53 231 248 181 181
query54 589 561 502 502
query55 88 86 83 83
query56 316 303 290 290
query57 1133 1157 1084 1084
query58 259 254 260 254
query59 2595 2612 2571 2571
query60 324 320 309 309
query61 138 163 129 129
query62 803 760 679 679
query63 218 192 193 192
query64 4345 990 666 666
query65 4324 4295 4244 4244
query66 1142 400 329 329
query67 15776 15318 15167 15167
query68 7568 887 510 510
query69 470 310 261 261
query70 1161 1166 1057 1057
query71 404 323 313 313
query72 5594 4705 4669 4669
query73 637 569 343 343
query74 9031 9122 8892 8892
query75 3153 3221 2692 2692
query76 3180 1170 754 754
query77 493 380 282 282
query78 9221 9424 8660 8660
query79 1339 817 574 574
query80 647 564 473 473
query81 494 257 220 220
query82 411 132 103 103
query83 267 265 259 259
query84 255 108 93 93
query85 767 409 281 281
query86 332 297 296 296
query87 4334 4385 4358 4358
query88 2871 2221 2233 2221
query89 386 318 294 294
query90 1744 214 211 211
query91 151 148 115 115
query92 62 62 56 56
query93 1089 949 591 591
query94 619 385 316 316
query95 373 297 293 293
query96 498 559 279 279
query97 3121 3217 3109 3109
query98 233 202 202 202
query99 1302 1403 1275 1275
Total cold run time: 268806 ms
Total hot run time: 184516 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2834947272
<details>
<summary>ClickBench: <b>Total hot run time: 29.17 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 787c34b286f794d7f261bb66d7823155b84c9868, data reload: false
query1 0.04 0.03 0.03
query2 0.12 0.10 0.11
query3 0.25 0.19 0.20
query4 1.58 0.19 0.20
query5 0.59 0.59 0.60
query6 1.19 0.71 0.73
query7 0.03 0.02 0.02
query8 0.04 0.04 0.04
query9 0.57 0.52 0.52
query10 0.57 0.58 0.57
query11 0.16 0.10 0.11
query12 0.14 0.11 0.11
query13 0.62 0.60 0.59
query14 0.78 0.82 0.82
query15 0.87 0.85 0.86
query16 0.39 0.38 0.38
query17 1.06 1.05 1.03
query18 0.21 0.19 0.19
query19 1.91 1.77 1.78
query20 0.01 0.01 0.02
query21 15.40 0.92 0.56
query22 0.77 1.17 0.73
query23 14.85 1.39 0.64
query24 6.78 1.66 0.69
query25 0.50 0.18 0.06
query26 0.54 0.17 0.13
query27 0.06 0.05 0.06
query28 10.07 0.92 0.43
query29 12.54 4.03 3.34
query30 0.26 0.09 0.06
query31 2.82 0.61 0.38
query32 3.23 0.55 0.46
query33 3.15 3.02 3.02
query34 15.85 5.14 4.50
query35 4.56 4.52 4.50
query36 0.67 0.50 0.49
query37 0.09 0.07 0.06
query38 0.05 0.05 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.14
query41 0.08 0.03 0.02
query42 0.03 0.03 0.02
query43 0.04 0.03 0.02
Total cold run time: 103.67 s
Total hot run time: 29.17 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2900686478
<details>
<summary>ClickBench: <b>Total hot run time: 29.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd9a107d966f7614132b60ddf52bb5c27d8ae0ff, data reload: false
query1 0.05 0.04 0.03
query2 0.11 0.10 0.11
query3 0.26 0.19 0.19
query4 1.60 0.19 0.19
query5 0.44 0.43 0.43
query6 1.15 0.65 0.66
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.58 0.51 0.52
query10 0.57 0.57 0.57
query11 0.16 0.11 0.10
query12 0.15 0.11 0.11
query13 0.60 0.60 0.60
query14 0.80 0.80 0.81
query15 0.87 0.86 0.85
query16 0.38 0.39 0.39
query17 1.02 1.08 1.05
query18 0.23 0.21 0.21
query19 1.93 1.81 1.84
query20 0.01 0.01 0.01
query21 15.41 0.91 0.55
query22 0.77 1.23 0.79
query23 14.74 1.39 0.64
query24 6.84 1.88 0.51
query25 0.51 0.12 0.12
query26 0.70 0.17 0.15
query27 0.05 0.04 0.05
query28 9.12 0.89 0.44
query29 12.58 4.03 3.45
query30 0.26 0.10 0.07
query31 2.82 0.59 0.39
query32 3.23 0.56 0.47
query33 3.03 3.07 3.05
query34 15.81 5.08 4.45
query35 4.56 4.48 4.49
query36 0.69 0.50 0.50
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.13 0.13
query41 0.09 0.03 0.03
query42 0.04 0.02 0.03
query43 0.04 0.03 0.04
Total cold run time: 102.6 s
Total hot run time: 29.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2893876761
<details>
<summary>TPC-DS: <b>Total hot run time: 185958 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 077b5b29596ba046f9e36350c68df03dd6caf0aa, data reload: false
query1 1007 474 498 474
query2 6564 1821 1783 1783
query3 6757 236 232 232
query4 26517 23454 23550 23454
query5 4372 633 465 465
query6 297 204 183 183
query7 4627 504 306 306
query8 322 268 256 256
query9 8626 2623 2648 2623
query10 501 337 290 290
query11 15887 15045 14962 14962
query12 165 113 111 111
query13 1646 529 421 421
query14 9237 6021 6172 6021
query15 208 189 169 169
query16 7175 655 469 469
query17 1188 718 578 578
query18 1979 401 285 285
query19 192 187 155 155
query20 119 123 115 115
query21 217 131 108 108
query22 4265 4171 4291 4171
query23 33947 33146 33320 33146
query24 8472 2420 2368 2368
query25 576 494 428 428
query26 1249 274 162 162
query27 2763 513 350 350
query28 4364 2128 2102 2102
query29 814 576 470 470
query30 288 228 199 199
query31 922 831 712 712
query32 80 68 68 68
query33 584 395 350 350
query34 817 859 519 519
query35 799 809 741 741
query36 956 999 911 911
query37 112 102 82 82
query38 4096 4089 4113 4089
query39 1467 1393 1442 1393
query40 222 128 116 116
query41 66 59 60 59
query42 137 120 113 113
query43 498 516 478 478
query44 1323 823 886 823
query45 175 174 167 167
query46 868 1026 633 633
query47 1776 1810 1764 1764
query48 392 413 324 324
query49 809 546 444 444
query50 659 688 424 424
query51 4079 4174 4140 4140
query52 112 104 99 99
query53 227 247 188 188
query54 577 563 503 503
query55 87 83 88 83
query56 329 296 278 278
query57 1144 1179 1090 1090
query58 268 263 261 261
query59 2622 2699 2503 2503
query60 333 327 300 300
query61 129 126 130 126
query62 792 726 646 646
query63 224 184 195 184
query64 4400 1011 668 668
query65 4307 4292 4263 4263
query66 1149 416 321 321
query67 16207 15802 15577 15577
query68 8896 899 524 524
query69 464 346 265 265
query70 1220 1096 1110 1096
query71 468 322 297 297
query72 5550 4650 4619 4619
query73 722 578 360 360
query74 8849 9148 8894 8894
query75 4194 3240 2716 2716
query76 3671 1187 768 768
query77 797 378 288 288
query78 9260 9373 8707 8707
query79 1917 887 570 570
query80 633 521 445 445
query81 470 256 220 220
query82 216 128 100 100
query83 289 256 241 241
query84 297 107 89 89
query85 718 323 284 284
query86 361 299 287 287
query87 4531 4368 4342 4342
query88 2886 2366 2299 2299
query89 401 315 279 279
query90 2093 210 204 204
query91 144 150 113 113
query92 71 66 59 59
query93 1347 978 587 587
query94 666 404 287 287
query95 376 299 290 290
query96 507 567 283 283
query97 2687 2778 2633 2633
query98 221 217 204 204
query99 1425 1423 1312 1312
Total cold run time: 274432 ms
Total hot run time: 185958 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2898546248
<details>
<summary>ClickBench: <b>Total hot run time: 29.31 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8398b2cd745248ad60a0a654105ed3b53b02bf7b, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.11 0.11
query3 0.26 0.18 0.20
query4 1.60 0.20 0.18
query5 0.44 0.46 0.44
query6 1.19 0.66 0.69
query7 0.03 0.02 0.02
query8 0.04 0.04 0.04
query9 0.60 0.52 0.53
query10 0.57 0.57 0.57
query11 0.16 0.10 0.11
query12 0.16 0.12 0.11
query13 0.61 0.59 0.60
query14 0.79 0.79 0.80
query15 0.87 0.88 0.85
query16 0.39 0.39 0.38
query17 1.03 1.05 1.05
query18 0.23 0.21 0.21
query19 1.99 1.91 1.77
query20 0.02 0.01 0.01
query21 15.40 0.91 0.54
query22 0.77 1.20 0.66
query23 14.92 1.38 0.65
query24 6.85 2.00 0.92
query25 0.48 0.32 0.07
query26 0.43 0.16 0.15
query27 0.06 0.06 0.05
query28 10.48 0.86 0.46
query29 12.59 4.00 3.40
query30 0.26 0.08 0.07
query31 2.82 0.61 0.38
query32 3.23 0.54 0.47
query33 3.01 3.07 3.06
query34 15.69 5.11 4.50
query35 4.50 4.55 4.53
query36 0.66 0.50 0.48
query37 0.08 0.06 0.06
query38 0.04 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.15 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.02
Total cold run time: 103.77 s
Total hot run time: 29.31 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2875975407
<details>
<summary>ClickBench: <b>Total hot run time: 28.76 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d39a11d88dea7be0865d4d4061a50ab3d8b34a69, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.10 0.11
query3 0.26 0.20 0.19
query4 1.59 0.20 0.11
query5 0.56 0.56 0.55
query6 1.20 0.71 0.72
query7 0.03 0.01 0.02
query8 0.04 0.03 0.04
query9 0.56 0.53 0.52
query10 0.59 0.58 0.57
query11 0.16 0.11 0.11
query12 0.14 0.12 0.12
query13 0.62 0.59 0.59
query14 0.78 0.80 0.81
query15 0.86 0.84 0.85
query16 0.39 0.38 0.38
query17 1.02 1.04 1.08
query18 0.23 0.21 0.21
query19 1.96 1.82 1.86
query20 0.02 0.01 0.01
query21 15.38 0.93 0.56
query22 0.75 1.14 0.62
query23 15.04 1.39 0.61
query24 6.78 2.38 0.42
query25 0.29 0.22 0.10
query26 0.68 0.16 0.14
query27 0.05 0.05 0.05
query28 9.06 0.93 0.44
query29 12.57 3.98 3.35
query30 0.25 0.09 0.07
query31 2.83 0.60 0.38
query32 3.23 0.54 0.46
query33 3.03 3.11 3.09
query34 15.82 5.08 4.45
query35 4.46 4.55 4.47
query36 0.68 0.49 0.48
query37 0.08 0.06 0.07
query38 0.05 0.04 0.04
query39 0.04 0.02 0.03
query40 0.18 0.14 0.14
query41 0.08 0.03 0.02
query42 0.04 0.02 0.03
query43 0.04 0.03 0.03
Total cold run time: 102.59 s
Total hot run time: 28.76 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2903972751
<details>
<summary>TPC-DS: <b>Total hot run time: 192173 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed7ada88affe7e1c21839f38dbc7af718c33487d, data reload: false
query1 1410 1104 1052 1052
query2 6225 1824 1824 1824
query3 11033 4467 4509 4467
query4 54948 25793 22903 22903
query5 5155 504 460 460
query6 380 217 192 192
query7 4902 500 289 289
query8 325 258 240 240
query9 6224 2652 2641 2641
query10 423 322 294 294
query11 15171 14928 14733 14733
query12 162 116 121 116
query13 1114 551 411 411
query14 10310 6317 6348 6317
query15 211 214 188 188
query16 7108 673 515 515
query17 1110 779 628 628
query18 1600 424 328 328
query19 214 210 177 177
query20 129 133 126 126
query21 229 130 121 121
query22 4480 4648 4280 4280
query23 34311 33602 33619 33602
query24 6563 2449 2515 2449
query25 459 499 413 413
query26 740 285 152 152
query27 2297 521 358 358
query28 2940 2186 2165 2165
query29 590 581 438 438
query30 283 226 188 188
query31 824 814 736 736
query32 72 67 64 64
query33 461 367 355 355
query34 808 876 526 526
query35 793 852 739 739
query36 969 1006 897 897
query37 121 108 95 95
query38 4349 4279 4228 4228
query39 1529 1493 1478 1478
query40 226 124 111 111
query41 63 60 54 54
query42 127 123 129 123
query43 507 497 492 492
query44 1417 876 861 861
query45 185 183 180 180
query46 884 1059 653 653
query47 1832 1921 1856 1856
query48 431 465 321 321
query49 703 556 453 453
query50 717 713 404 404
query51 4206 4316 4192 4192
query52 118 116 107 107
query53 243 270 192 192
query54 597 585 529 529
query55 88 84 88 84
query56 303 307 301 301
query57 1219 1207 1124 1124
query58 280 271 261 261
query59 2711 2792 2685 2685
query60 335 340 319 319
query61 143 127 153 127
query62 735 787 689 689
query63 234 190 193 190
query64 2000 1030 688 688
query65 4320 4224 4252 4224
query66 751 400 305 305
query67 16041 15731 15517 15517
query68 7167 835 538 538
query69 549 309 273 273
query70 1204 1105 1099 1099
query71 501 323 306 306
query72 5952 4871 4988 4871
query73 1356 701 359 359
query74 9308 8808 8913 8808
query75 3799 3204 2687 2687
query76 4259 1200 790 790
query77 607 369 283 283
query78 9319 10067 8696 8696
query79 1769 819 580 580
query80 599 530 459 459
query81 480 259 222 222
query82 406 130 98 98
query83 260 263 235 235
query84 294 107 91 91
query85 775 340 289 289
query86 366 295 308 295
query87 4441 4480 4308 4308
query88 3021 2323 2329 2323
query89 423 320 281 281
query90 1949 228 223 223
query91 140 145 131 131
query92 80 60 55 55
query93 1093 964 584 584
query94 715 425 318 318
query95 378 302 294 294
query96 504 572 279 279
query97 2773 2737 2668 2668
query98 225 203 206 203
query99 1505 1391 1300 1300
Total cold run time: 298605 ms
Total hot run time: 192173 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "feiniaofeiafei (via GitHub)" <gi...@apache.org>.
feiniaofeiafei commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2085871023
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalUnion.java:
##########
@@ -206,7 +213,53 @@ public void computeUnique(DataTrait.Builder builder) {
@Override
public void computeUniform(DataTrait.Builder builder) {
- // don't propagate uniform slots
+ CascadesContext tempCascadeContext = CascadesContext.initContext(
+ ConnectContext.get().getStatementContext(), this, PhysicalProperties.ANY);
+ ExpressionRewriteContext context = new ExpressionRewriteContext(tempCascadeContext);
+ for (int i = 0; i < getOutputs().size(); i++) {
+ Optional<Literal> value = Optional.empty();
+ if (!constantExprsList.isEmpty()) {
+ value = ExpressionUtils.checkConstantExpr(constantExprsList.get(0).get(i), context);
+ if (!value.isPresent()) {
+ continue;
+ }
+ final int fi = i;
+ Literal literal = value.get();
+ if (constantExprsList.stream()
+ .map(exprs -> ExpressionUtils.checkConstantExpr(exprs.get(fi), context))
+ .anyMatch(val -> !val.isPresent() || !val.get().equals(literal))) {
+ continue;
+ }
+ }
+ for (int childIdx = 0; childIdx < children.size(); childIdx++) {
+ // TODO: use originOutputs = child(childIdx).getOutput() ?
+ List<? extends Slot> originOutputs = regularChildrenOutputs.get(childIdx);
+ Slot slot = originOutputs.get(i);
+ Optional<Expression> childValue = child(childIdx).getLogicalProperties()
+ .getTrait().getUniformValue(slot);
+ if (childValue == null || !childValue.isPresent() || !childValue.get().isConstant()) {
+ value = Optional.empty();
+ break;
+ }
+ Expression childExpr = childValue.get();
+ if (!childExpr.isLiteral()) {
+ childExpr = FoldConstantRule.evaluate(childExpr, context);
+ if (!childExpr.isLiteral()) {
+ continue;
Review Comment:
this continue may lead to wrong result, should use value=Optional.empty(); break; for exmaple:select c1 from (
select cast('abc' as char) c1 from test_skew_window union all select 'bcd'from test_skew_window) t group by c1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3117214778
<details>
<summary>TPC-DS: <b>Total hot run time: 186622 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3f0b61e7e8cc8de1b85cb8c4ac561353c78b15d, data reload: false
query1 1013 386 420 386
query2 6536 1653 1673 1653
query3 6734 220 227 220
query4 26329 23784 23152 23152
query5 4345 620 497 497
query6 309 215 201 201
query7 4619 501 303 303
query8 269 229 217 217
query9 8600 2883 2888 2883
query10 468 315 281 281
query11 15912 15061 14775 14775
query12 160 117 122 117
query13 1660 539 428 428
query14 9455 5954 5876 5876
query15 201 188 162 162
query16 7507 604 492 492
query17 1210 749 587 587
query18 2024 424 322 322
query19 199 191 170 170
query20 135 116 115 115
query21 212 123 108 108
query22 4472 4202 4126 4126
query23 33957 33128 32887 32887
query24 8056 2368 2389 2368
query25 579 463 413 413
query26 935 273 157 157
query27 2710 520 349 349
query28 4388 2232 2206 2206
query29 689 553 471 471
query30 289 218 192 192
query31 911 801 703 703
query32 78 75 69 69
query33 537 371 336 336
query34 790 854 523 523
query35 779 809 755 755
query36 975 1015 942 942
query37 120 106 88 88
query38 4121 4064 4013 4013
query39 1474 1393 1384 1384
query40 213 130 116 116
query41 65 59 56 56
query42 121 107 110 107
query43 510 506 450 450
query44 1349 862 869 862
query45 179 175 172 172
query46 840 996 645 645
query47 1772 1809 1716 1716
query48 381 419 307 307
query49 678 489 403 403
query50 641 693 390 390
query51 5558 5571 5388 5388
query52 125 106 102 102
query53 232 267 186 186
query54 588 601 532 532
query55 92 88 90 88
query56 319 323 305 305
query57 1209 1188 1120 1120
query58 280 274 280 274
query59 2485 2698 2546 2546
query60 351 353 323 323
query61 127 119 123 119
query62 781 732 655 655
query63 227 190 194 190
query64 3559 1009 685 685
query65 4246 4167 4217 4167
query66 948 427 326 326
query67 15741 15761 15543 15543
query68 7958 905 580 580
query69 477 315 282 282
query70 1177 1153 1144 1144
query71 456 325 316 316
query72 5355 4652 4696 4652
query73 724 602 358 358
query74 9242 9167 8660 8660
query75 3883 3097 2603 2603
query76 3603 1161 715 715
query77 795 388 315 315
query78 9521 9868 8960 8960
query79 2196 840 599 599
query80 625 546 483 483
query81 483 246 222 222
query82 454 135 110 110
query83 255 260 239 239
query84 246 103 80 80
query85 868 356 324 324
query86 338 310 313 310
query87 4375 4355 4313 4313
query88 3305 2248 2243 2243
query89 393 322 288 288
query90 1946 233 221 221
query91 141 147 116 116
query92 85 67 68 67
query93 1153 989 652 652
query94 695 410 300 300
query95 478 320 311 311
query96 486 581 281 281
query97 2705 2705 2662 2662
query98 240 220 212 212
query99 1661 1422 1304 1304
Total cold run time: 273693 ms
Total hot run time: 186622 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3026925170
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2804541239
<details>
<summary>TPC-DS: <b>Total hot run time: 191330 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2930d988d2c992f3fe1a6254e06d831d20a0a171, data reload: false
============================================
query1 1396 1103 1076 1076
query2 6176 1879 1908 1879
query3 11135 4711 4564 4564
query4 25518 23685 22950 22950
query5 3674 638 500 500
query6 301 199 199 199
query7 3987 492 277 277
query8 295 241 231 231
query9 8464 2572 2616 2572
query10 471 322 281 281
query11 15866 15370 14807 14807
query12 183 114 112 112
query13 1564 528 417 417
query14 9215 6072 6122 6072
query15 206 191 170 170
query16 7283 649 474 474
query17 1200 739 598 598
query18 2002 429 322 322
query19 200 201 167 167
query20 130 127 126 126
query21 207 131 108 108
query22 4480 4516 4476 4476
query23 34278 33740 33460 33460
query24 8642 2421 2404 2404
query25 515 485 398 398
query26 718 266 161 161
query27 2904 498 327 327
query28 4657 2457 2424 2424
query29 649 568 443 443
query30 278 219 193 193
query31 868 834 755 755
query32 76 84 70 70
query33 537 352 328 328
query34 803 883 510 510
query35 822 841 777 777
query36 975 1000 895 895
query37 133 105 80 80
query38 4226 4246 4246 4246
query39 1497 1472 1460 1460
query40 212 119 106 106
query41 62 64 53 53
query42 125 118 112 112
query43 502 495 481 481
query44 1305 813 844 813
query45 188 175 175 175
query46 851 1024 651 651
query47 1845 1859 1760 1760
query48 383 413 311 311
query49 query50 653 691 413 413
query51 4208 4264 4230 4230
query52 110 101 100 100
query53 222 262 183 183
query54 577 591 505 505
query55 85 88 87 87
query56 320 297 296 296
query57 1162 1179 1120 1120
query58 272 278 255 255
query59 2757 2747 2733 2733
query60 332 315 303 303
query61 136 124 149 124
query62 774 756 664 664
query63 218 196 182 182
query64 3078 1077 707 707
query65 4377 4293 4394 4293
query66 896 421 304 304
query67 16102 15701 15472 15472
query68 8950 888 509 509
query69 496 290 265 265
query70 1248 1090 1046 1046
query71 463 317 295 295
query72 5176 4682 4614 4614
query73 701 565 339 339
query74 8996 9135 8953 8953
query75 4185 3229 2722 2722
query76 3658 1181 742 742
query77 777 383 278 278
query78 8960 9035 8301 8301
query79 2092 927 574 574
query80 564 517 439 439
query81 473 260 226 226
query82 485 131 102 102
query83 257 296 233 233
query84 248 105 83 83
query85 828 320 264 264
query86 375 304 283 283
query87 4397 4442 4373 4373
query88 3574 2276 2231 2231
query89 400 321 286 286
query90 1769 209 212 209
query91 143 144 110 110
query92 84 63 60 60
query93 1718 979 578 578
query94 613 417 308 308
query95 373 296 279 279
query96 487 564 273 273
query97 3156 3241 3180 3180
query98 226 210 197 197
query99 1325 1388 1236 1236
Total cold run time: 275480 ms
Total hot run time: 191330 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2798453521
<details>
<summary>TPC-H: <b>Total hot run time: 34121 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 679002894d5fb311f725de08d1ffda48e95234fc, data reload: false
------ Round 1 ----------------------------------
q1 25911 5023 5016 5016
q2 2071 268 180 180
q3 10398 1243 689 689
q4 10228 1011 540 540
q5 7539 2370 2324 2324
q6 186 167 132 132
q7 933 733 631 631
q8 9304 1216 1085 1085
q9 6827 5093 5127 5093
q10 6876 2310 1873 1873
q11 494 280 278 278
q12 349 344 222 222
q13 17793 3646 3084 3084
q14 227 227 206 206
q15 542 507 495 495
q16 642 611 574 574
q17 577 853 357 357
q18 7541 7254 7089 7089
q19 1442 949 577 577
q20 339 328 227 227
q21 4027 3405 2480 2480
q22 1063 1024 969 969
Total cold run time: 115309 ms
Total hot run time: 34121 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5149 5121 5046 5046
q2 244 329 235 235
q3 2211 2705 2345 2345
q4 1463 1828 1449 1449
q5 4505 4357 4386 4357
q6 217 177 138 138
q7 2009 1915 1737 1737
q8 2565 2623 2525 2525
q9 7261 7270 7061 7061
q10 2967 3159 2766 2766
q11 572 519 496 496
q12 712 775 627 627
q13 3504 3894 3348 3348
q14 311 318 272 272
q15 520 481 494 481
q16 660 688 650 650
q17 1152 1492 1404 1404
q18 7718 7668 7576 7576
q19 816 778 837 778
q20 1927 1951 1804 1804
q21 5218 4967 4823 4823
q22 1094 1062 1023 1023
Total cold run time: 52795 ms
Total hot run time: 50941 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2804403518
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3073772460
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3073967809
<details>
<summary>TPC-DS: <b>Total hot run time: 187193 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8155bdac55ae0d1587c99381e8f178452066e663, data reload: false
query1 1118 397 411 397
query2 6510 1722 1701 1701
query3 6747 221 224 221
query4 26620 23780 23448 23448
query5 4378 593 436 436
query6 302 224 215 215
query7 4618 490 282 282
query8 274 214 217 214
query9 8642 2622 2617 2617
query10 463 319 272 272
query11 16107 15009 14787 14787
query12 153 102 102 102
query13 1646 538 400 400
query14 9236 6118 6070 6070
query15 214 201 166 166
query16 7405 424 250 250
query17 1348 688 569 569
query18 1994 389 301 301
query19 192 192 158 158
query20 121 120 117 117
query21 214 124 108 108
query22 4338 4232 4137 4137
query23 33923 32918 33117 32918
query24 8391 2348 2355 2348
query25 535 470 412 412
query26 1237 265 143 143
query27 2761 503 344 344
query28 4338 2134 2109 2109
query29 738 537 445 445
query30 292 224 191 191
query31 902 829 703 703
query32 72 65 64 64
query33 569 356 281 281
query34 798 863 517 517
query35 603 651 578 578
query36 979 1038 926 926
query37 110 101 90 90
query38 4118 4214 4091 4091
query39 1509 1467 1444 1444
query40 215 123 110 110
query41 69 60 60 60
query42 120 106 112 106
query43 502 527 490 490
query44 1321 822 809 809
query45 177 173 165 165
query46 832 1025 619 619
query47 1757 1823 1725 1725
query48 380 424 316 316
query49 734 476 386 386
query50 635 679 407 407
query51 5523 5390 5489 5390
query52 116 103 96 96
query53 217 252 186 186
query54 558 554 488 488
query55 85 80 80 80
query56 321 320 288 288
query57 1225 1226 1118 1118
query58 278 252 256 252
query59 2556 2611 2606 2606
query60 324 335 322 322
query61 129 132 129 129
query62 803 717 650 650
query63 218 184 193 184
query64 4347 1167 835 835
query65 4257 4185 4200 4185
query66 1106 410 341 341
query67 15837 15729 15783 15729
query68 5596 904 542 542
query69 527 318 271 271
query70 1252 1167 1164 1164
query71 403 323 307 307
query72 5599 4674 5022 4674
query73 685 649 353 353
query74 8882 9205 8955 8955
query75 3188 3199 2699 2699
query76 3218 1140 711 711
query77 496 401 311 311
query78 10020 10332 9429 9429
query79 1519 884 582 582
query80 690 520 448 448
query81 467 262 222 222
query82 182 122 96 96
query83 250 251 237 237
query84 248 103 94 94
query85 760 379 323 323
query86 335 347 277 277
query87 4425 4539 4387 4387
query88 2862 2301 2288 2288
query89 378 319 282 282
query90 1766 211 205 205
query91 136 144 120 120
query92 69 62 59 59
query93 1034 974 589 589
query94 628 315 199 199
query95 371 288 277 277
query96 494 560 275 275
query97 2688 2801 2653 2653
query98 241 217 201 201
query99 1367 1422 1319 1319
Total cold run time: 269933 ms
Total hot run time: 187193 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3073986284
<details>
<summary>ClickBench: <b>Total hot run time: 32.2 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8155bdac55ae0d1587c99381e8f178452066e663, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.25 0.07 0.07
query4 1.60 0.10 0.11
query5 0.44 0.42 0.40
query6 1.16 0.66 0.66
query7 0.03 0.02 0.02
query8 0.05 0.04 0.03
query9 0.61 0.50 0.50
query10 0.56 0.57 0.56
query11 0.16 0.11 0.11
query12 0.15 0.12 0.13
query13 0.63 0.62 0.61
query14 0.80 0.82 0.83
query15 0.90 0.86 0.90
query16 0.39 0.38 0.38
query17 1.08 1.04 1.08
query18 0.23 0.22 0.21
query19 1.97 1.88 1.86
query20 0.02 0.01 0.01
query21 15.38 0.95 0.54
query22 0.77 1.23 0.71
query23 14.87 1.39 0.62
query24 7.53 1.49 0.58
query25 0.45 0.18 0.06
query26 0.59 0.17 0.14
query27 0.06 0.05 0.06
query28 9.02 0.85 0.46
query29 12.56 3.99 3.33
query30 3.04 2.98 2.94
query31 2.83 0.60 0.39
query32 3.22 0.57 0.48
query33 3.11 3.07 3.15
query34 16.09 5.34 4.79
query35 4.91 4.84 4.77
query36 0.69 0.52 0.50
query37 0.09 0.07 0.07
query38 0.05 0.04 0.04
query39 0.03 0.03 0.02
query40 0.17 0.14 0.13
query41 0.08 0.03 0.02
query42 0.04 0.03 0.03
query43 0.03 0.04 0.03
Total cold run time: 106.76 s
Total hot run time: 32.2 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833067114
<details>
<summary>TPC-H: <b>Total hot run time: 33919 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit acef217490988ac1b4f5bb9e8a5496b5225ff8d3, data reload: false
------ Round 1 ----------------------------------
q1 26184 5052 5121 5052
q2 2084 295 185 185
q3 10463 1266 686 686
q4 10227 991 510 510
q5 7729 2359 2359 2359
q6 272 163 129 129
q7 923 731 615 615
q8 9323 1290 1002 1002
q9 7464 5147 5077 5077
q10 6815 2308 1878 1878
q11 496 292 276 276
q12 347 346 221 221
q13 17762 3685 3097 3097
q14 228 231 222 222
q15 537 495 481 481
q16 443 452 400 400
q17 609 858 382 382
q18 7691 7119 7094 7094
q19 1721 947 554 554
q20 336 339 221 221
q21 4089 3450 2489 2489
q22 1095 1037 989 989
Total cold run time: 116838 ms
Total hot run time: 33919 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5124 5109 5059 5059
q2 244 330 225 225
q3 2169 2642 2310 2310
q4 1352 1825 1513 1513
q5 4577 4440 4333 4333
q6 213 172 132 132
q7 1971 1924 1727 1727
q8 2571 2545 2504 2504
q9 7254 7238 7163 7163
q10 3022 3222 2705 2705
q11 569 531 502 502
q12 634 747 605 605
q13 3478 3843 3305 3305
q14 292 295 272 272
q15 524 461 472 461
q16 465 515 478 478
q17 1135 1566 1360 1360
q18 7523 7601 7337 7337
q19 804 813 807 807
q20 1974 2018 1911 1911
q21 5120 4556 4557 4556
q22 1093 1032 1015 1015
Total cold run time: 52108 ms
Total hot run time: 50280 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2826717291
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2827310577
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2829991750
<details>
<summary>TPC-DS: <b>Total hot run time: 192710 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c93fc833a1a8da818b939c19fbf70373fb15231, data reload: false
query1 1417 1104 1050 1050
query2 6161 1915 1871 1871
query3 11365 4911 4679 4679
query4 25641 23801 23455 23455
query5 4846 653 502 502
query6 311 212 202 202
query7 3992 484 284 284
query8 283 238 234 234
query9 8511 2528 2537 2528
query10 480 319 284 284
query11 16244 15027 14745 14745
query12 163 115 115 115
query13 1570 558 369 369
query14 9441 6148 6049 6049
query15 217 189 191 189
query16 7581 641 432 432
query17 1100 746 587 587
query18 2045 409 305 305
query19 193 199 188 188
query20 133 130 122 122
query21 206 128 112 112
query22 4472 4526 4412 4412
query23 34527 33803 33593 33593
query24 8539 2447 2414 2414
query25 569 489 445 445
query26 1316 276 152 152
query27 2835 501 340 340
query28 5018 2160 2133 2133
query29 699 557 442 442
query30 274 232 199 199
query31 900 847 766 766
query32 75 67 68 67
query33 549 380 319 319
query34 801 860 534 534
query35 796 849 808 808
query36 993 985 934 934
query37 115 100 74 74
query38 4307 4174 4171 4171
query39 1480 1447 1438 1438
query40 211 118 106 106
query41 55 59 55 55
query42 112 106 107 106
query43 527 514 481 481
query44 1312 814 810 810
query45 184 179 169 169
query46 849 1036 652 652
query47 1870 1891 1806 1806
query48 382 423 316 316
query49 772 523 445 445
query50 648 687 412 412
query51 4259 4242 4155 4155
query52 113 106 104 104
query53 272 272 186 186
query54 591 567 519 519
query55 85 78 83 78
query56 306 308 311 308
query57 1201 1208 1178 1178
query58 268 264 264 264
query59 2734 2967 2840 2840
query60 339 331 309 309
query61 136 126 124 124
query62 753 783 680 680
query63 237 189 192 189
query64 3986 1098 787 787
query65 4507 4369 4374 4369
query66 1047 422 339 339
query67 15811 15607 15276 15276
query68 8355 901 515 515
query69 479 307 278 278
query70 1244 1119 1109 1109
query71 462 329 312 312
query72 5495 4744 4819 4744
query73 666 648 345 345
query74 9041 9101 8797 8797
query75 3919 3228 2719 2719
query76 3593 1192 765 765
query77 809 371 283 283
query78 9498 9369 8726 8726
query79 2708 812 583 583
query80 695 501 466 466
query81 499 257 222 222
query82 470 123 103 103
query83 282 251 232 232
query84 293 106 89 89
query85 747 323 280 280
query86 377 324 287 287
query87 4352 4380 4266 4266
query88 3629 2260 2246 2246
query89 399 324 283 283
query90 1851 212 210 210
query91 140 138 111 111
query92 74 62 60 60
query93 2315 940 578 578
query94 659 418 309 309
query95 379 293 284 284
query96 490 573 277 277
query97 3182 3232 3120 3120
query98 228 203 210 203
query99 1411 1396 1302 1302
Total cold run time: 282473 ms
Total hot run time: 192710 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833104300
<details>
<summary>ClickBench: <b>Total hot run time: 29.36 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit acef217490988ac1b4f5bb9e8a5496b5225ff8d3, data reload: false
query1 0.04 0.03 0.03
query2 0.12 0.10 0.10
query3 0.25 0.19 0.18
query4 1.60 0.19 0.20
query5 0.56 0.54 0.57
query6 1.18 0.71 0.71
query7 0.03 0.02 0.02
query8 0.04 0.04 0.04
query9 0.58 0.51 0.51
query10 0.58 0.57 0.56
query11 0.15 0.11 0.11
query12 0.15 0.11 0.11
query13 0.61 0.59 0.59
query14 1.19 1.17 1.22
query15 0.87 0.85 0.86
query16 0.38 0.39 0.39
query17 1.01 1.05 1.03
query18 0.21 0.19 0.20
query19 1.94 1.81 1.79
query20 0.01 0.01 0.02
query21 15.41 0.91 0.55
query22 0.75 1.20 0.71
query23 14.89 1.37 0.64
query24 7.20 1.54 0.58
query25 0.52 0.24 0.07
query26 0.71 0.17 0.13
query27 0.05 0.06 0.05
query28 9.51 0.87 0.43
query29 12.51 3.89 3.27
query30 0.26 0.09 0.08
query31 2.85 0.57 0.39
query32 3.23 0.55 0.47
query33 3.01 3.06 3.08
query34 15.88 5.11 4.53
query35 4.53 4.55 4.49
query36 0.66 0.50 0.47
query37 0.09 0.06 0.07
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.14 0.13
query41 0.08 0.03 0.02
query42 0.04 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 103.97 s
Total hot run time: 29.36 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838096038
<details>
<summary>TPC-H: <b>Total hot run time: 34668 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 135312542e23c5083a8d9116bbce26d9e4f6f1c0, data reload: false
------ Round 1 ----------------------------------
q1 26637 4992 5001 4992
q2 2072 285 184 184
q3 10423 1254 722 722
q4 10231 1002 510 510
q5 7535 2380 2379 2379
q6 188 163 130 130
q7 926 757 637 637
q8 9320 1304 1037 1037
q9 6852 5062 5107 5062
q10 6871 2301 1907 1907
q11 506 297 271 271
q12 351 366 211 211
q13 17795 3742 3129 3129
q14 233 233 231 231
q15 547 489 494 489
q16 443 427 382 382
q17 611 852 357 357
q18 7561 7231 7147 7147
q19 1583 966 569 569
q20 326 336 229 229
q21 4293 3414 3142 3142
q22 1056 993 951 951
Total cold run time: 116360 ms
Total hot run time: 34668 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5157 5067 5070 5067
q2 242 325 229 229
q3 2172 2670 2249 2249
q4 1374 1837 1464 1464
q5 4536 4431 4355 4355
q6 213 166 126 126
q7 1992 1908 1798 1798
q8 2581 2549 2526 2526
q9 7234 7281 7101 7101
q10 3038 3174 2762 2762
q11 570 538 498 498
q12 684 810 591 591
q13 3645 3941 3291 3291
q14 305 323 302 302
q15 522 489 470 470
q16 421 476 436 436
q17 1149 1548 1381 1381
q18 7777 7794 7522 7522
q19 789 828 876 828
q20 1970 2032 1863 1863
q21 5216 4786 4808 4786
q22 1097 1080 1013 1013
Total cold run time: 52684 ms
Total hot run time: 50658 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2827330417
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2872479938
<details>
<summary>TPC-DS: <b>Total hot run time: 186288 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea6e2dde86b7743e3f5550e1f93c4874995b9b05, data reload: false
query1 1013 468 482 468
query2 6574 1836 1806 1806
query3 6771 240 225 225
query4 26297 23346 23439 23346
query5 4338 601 500 500
query6 304 208 197 197
query7 4616 481 298 298
query8 308 245 233 233
query9 8577 2650 2664 2650
query10 511 329 273 273
query11 15710 15029 14843 14843
query12 154 109 114 109
query13 1665 547 419 419
query14 9139 6176 6157 6157
query15 222 189 181 181
query16 7169 610 483 483
query17 1223 740 598 598
query18 1987 409 309 309
query19 196 193 167 167
query20 123 123 116 116
query21 218 128 113 113
query22 4124 4239 4114 4114
query23 34152 33357 33327 33327
query24 8479 2445 2364 2364
query25 537 456 400 400
query26 1237 273 154 154
query27 2736 492 336 336
query28 4292 2135 2114 2114
query29 766 557 440 440
query30 284 218 188 188
query31 911 803 735 735
query32 77 69 65 65
query33 553 376 316 316
query34 813 858 532 532
query35 762 820 734 734
query36 975 990 877 877
query37 118 101 80 80
query38 4141 4137 4090 4090
query39 1461 1424 1424 1424
query40 209 119 115 115
query41 58 60 52 52
query42 127 117 108 108
query43 484 503 489 489
query44 1308 815 829 815
query45 187 173 167 167
query46 846 1021 619 619
query47 1743 1819 1743 1743
query48 385 432 309 309
query49 769 526 443 443
query50 665 707 407 407
query51 4105 4050 4038 4038
query52 105 109 98 98
query53 232 259 185 185
query54 596 581 501 501
query55 88 80 83 80
query56 318 297 307 297
query57 1144 1168 1106 1106
query58 272 289 266 266
query59 2562 2786 2584 2584
query60 339 326 300 300
query61 129 129 127 127
query62 802 755 678 678
query63 227 188 184 184
query64 4316 1009 731 731
query65 4344 4243 4243 4243
query66 1155 406 303 303
query67 15910 15727 15441 15441
query68 8178 893 514 514
query69 464 314 275 275
query70 1201 1109 1088 1088
query71 465 368 314 314
query72 5552 4735 4565 4565
query73 702 559 356 356
query74 8875 9182 8717 8717
query75 3799 3223 2761 2761
query76 3654 1186 757 757
query77 787 404 297 297
query78 9514 9485 8718 8718
query79 2163 812 588 588
query80 572 519 456 456
query81 494 259 231 231
query82 473 132 106 106
query83 261 257 238 238
query84 250 105 88 88
query85 803 400 277 277
query86 389 306 283 283
query87 4500 4502 4460 4460
query88 3651 2288 2274 2274
query89 408 317 280 280
query90 1848 208 208 208
query91 137 149 112 112
query92 76 65 57 57
query93 1621 952 585 585
query94 722 403 316 316
query95 376 294 283 283
query96 502 580 282 282
query97 3120 3233 3114 3114
query98 230 215 217 215
query99 1380 1379 1263 1263
Total cold run time: 274125 ms
Total hot run time: 186288 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833420700
<details>
<summary>TPC-H: <b>Total hot run time: 34001 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 623b77512221f6c2e11c5e755e77011eb36e2a01, data reload: false
------ Round 1 ----------------------------------
q1 26449 5150 5089 5089
q2 2092 297 200 200
q3 10505 1259 724 724
q4 10244 1014 538 538
q5 7757 2362 2357 2357
q6 176 162 135 135
q7 897 757 612 612
q8 9315 1279 1111 1111
q9 6796 5005 5026 5005
q10 6822 2302 1868 1868
q11 476 286 270 270
q12 334 369 203 203
q13 17783 3664 3105 3105
q14 227 229 207 207
q15 545 494 490 490
q16 472 443 398 398
q17 598 851 389 389
q18 7584 7339 7090 7090
q19 1203 935 540 540
q20 332 321 210 210
q21 4313 3432 2480 2480
q22 1085 1018 980 980
Total cold run time: 116005 ms
Total hot run time: 34001 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5113 5106 5501 5106
q2 237 320 231 231
q3 2130 2659 2281 2281
q4 1354 1797 1459 1459
q5 4484 4386 4378 4378
q6 217 166 128 128
q7 2023 1934 1790 1790
q8 2581 2560 2558 2558
q9 7290 7199 7179 7179
q10 2973 3184 2722 2722
q11 597 517 505 505
q12 668 781 595 595
q13 3506 3892 3374 3374
q14 290 288 263 263
q15 537 509 477 477
q16 476 512 479 479
q17 1164 1543 1412 1412
q18 7737 7538 7479 7479
q19 825 795 819 795
q20 1947 1998 1812 1812
q21 5100 4818 4770 4770
q22 1100 1105 1061 1061
Total cold run time: 52349 ms
Total hot run time: 50854 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2837990028
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2919111759
<details>
<summary>TPC-DS: <b>Total hot run time: 185635 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72063f534c073770ef22ff79f1cb2eb3c505b7e7, data reload: false
query1 1019 472 530 472
query2 6588 1810 1823 1810
query3 6753 229 227 227
query4 25999 23308 23466 23308
query5 5331 668 449 449
query6 308 203 200 200
query7 4646 480 291 291
query8 270 235 218 218
query9 8603 2617 2666 2617
query10 504 348 283 283
query11 15871 15082 14932 14932
query12 180 114 112 112
query13 1647 528 432 432
query14 10248 6134 6128 6128
query15 203 193 178 178
query16 7678 647 459 459
query17 1311 735 584 584
query18 2025 416 315 315
query19 200 201 168 168
query20 123 119 124 119
query21 221 129 111 111
query22 4138 4046 4038 4038
query23 34041 33038 33257 33038
query24 8290 2372 2370 2370
query25 524 452 386 386
query26 1235 274 150 150
query27 2716 504 352 352
query28 4299 2135 2106 2106
query29 750 555 488 488
query30 282 213 194 194
query31 914 835 729 729
query32 70 67 70 67
query33 585 372 318 318
query34 805 849 556 556
query35 817 824 720 720
query36 953 1007 871 871
query37 108 98 75 75
query38 4148 4141 4086 4086
query39 1471 1421 1409 1409
query40 214 118 107 107
query41 66 60 59 59
query42 118 117 110 110
query43 498 509 477 477
query44 1296 827 828 827
query45 181 184 174 174
query46 836 1021 619 619
query47 1752 1806 1726 1726
query48 385 420 314 314
query49 759 480 396 396
query50 646 685 404 404
query51 4057 4248 4126 4126
query52 104 107 100 100
query53 225 253 201 201
query54 577 568 498 498
query55 87 93 84 84
query56 318 324 297 297
query57 1125 1143 1100 1100
query58 267 263 256 256
query59 2492 2586 2557 2557
query60 330 329 309 309
query61 136 133 135 133
query62 808 724 694 694
query63 219 197 186 186
query64 4351 1053 676 676
query65 4254 4141 4131 4131
query66 1038 414 315 315
query67 15926 15857 15501 15501
query68 8138 903 531 531
query69 504 306 276 276
query70 1185 1100 1146 1100
query71 492 351 310 310
query72 5646 4656 4699 4656
query73 679 603 359 359
query74 8938 9172 8960 8960
query75 3865 3219 2697 2697
query76 3737 1178 736 736
query77 794 385 283 283
query78 9372 9455 8685 8685
query79 2211 875 572 572
query80 647 510 473 473
query81 491 259 230 230
query82 448 130 97 97
query83 257 252 240 240
query84 252 111 96 96
query85 739 335 294 294
query86 389 320 286 286
query87 4568 4433 4376 4376
query88 3642 2319 2328 2319
query89 386 320 290 290
query90 1953 214 215 214
query91 150 152 118 118
query92 72 68 64 64
query93 1793 941 589 589
query94 677 410 307 307
query95 380 305 284 284
query96 500 592 287 287
query97 2747 2765 2684 2684
query98 236 220 202 202
query99 1620 1403 1277 1277
Total cold run time: 276447 ms
Total hot run time: 185635 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2986164435
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2220971260
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java:
##########
@@ -259,6 +260,7 @@ public class Rewriter extends AbstractBatchJobExecutor {
bottomUp(new InlineLogicalView())
),
topic("Eliminate optimization",
+ // custom(RuleType.CONSTANT_PROPAGATION, ConstantPropagation::new),
Review Comment:
delete them
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3117365354
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3116574524
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3086576926
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209064420
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionNormalizationAndOptimization.java:
##########
@@ -21,17 +21,23 @@
import com.google.common.collect.ImmutableList;
+import java.util.List;
+
/** ExpressionNormalizationAndOptimization */
public class ExpressionNormalizationAndOptimization extends ExpressionRewrite {
/** ExpressionNormalizationAndOptimization */
- public ExpressionNormalizationAndOptimization() {
- super(new ExpressionRuleExecutor(
- ImmutableList.<ExpressionRewriteRule<ExpressionRewriteContext>>builder()
- .addAll(ExpressionNormalization.NORMALIZE_REWRITE_RULES)
- .addAll(ExpressionOptimization.OPTIMIZE_REWRITE_RULES)
- .addAll(ExpressionOptimization.ADD_RANGE)
- .build()
- ));
+ public ExpressionNormalizationAndOptimization(boolean addRange) {
+ super(new ExpressionRuleExecutor(buildRewriteRule(addRange)));
+ }
+
+ private static List<ExpressionRewriteRule<ExpressionRewriteContext>> buildRewriteRule(boolean addRange) {
+ ImmutableList.Builder<ExpressionRewriteRule<ExpressionRewriteContext>> builder = ImmutableList.builder();
+ builder.addAll(ExpressionNormalization.NORMALIZE_REWRITE_RULES)
+ .addAll(ExpressionOptimization.OPTIMIZE_REWRITE_RULES);
+ if (addRange) {
Review Comment:
add annotation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076525705
<details>
<summary>TPC-H: <b>Total hot run time: 33073 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 96e01840ac51fd74427a6987028ab866c386ed9f, data reload: false
------ Round 1 ----------------------------------
q1 17557 5182 5115 5115
q2 1965 272 179 179
q3 10303 1249 716 716
q4 10233 1006 533 533
q5 7496 2338 2301 2301
q6 178 159 128 128
q7 880 734 598 598
q8 9295 1289 1093 1093
q9 6865 5044 5102 5044
q10 6881 2366 1971 1971
q11 485 280 256 256
q12 339 355 203 203
q13 17761 3650 3102 3102
q14 242 227 207 207
q15 544 478 476 476
q16 415 419 369 369
q17 557 858 344 344
q18 7474 7157 7049 7049
q19 1221 951 542 542
q20 336 342 211 211
q21 3583 2531 2344 2344
q22 352 313 292 292
Total cold run time: 104962 ms
Total hot run time: 33073 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5125 5090 5057 5057
q2 239 323 221 221
q3 2188 2667 2267 2267
q4 1351 1796 1339 1339
q5 4179 4094 4402 4094
q6 214 180 127 127
q7 2082 1938 1860 1860
q8 2643 2543 2529 2529
q9 7299 7246 7240 7240
q10 3078 3312 2853 2853
q11 571 528 480 480
q12 706 764 670 670
q13 3715 4013 3296 3296
q14 276 311 272 272
q15 538 491 505 491
q16 463 492 456 456
q17 1153 1563 1375 1375
q18 7863 7675 7493 7493
q19 755 742 889 742
q20 1899 1965 1860 1860
q21 4796 4420 4249 4249
q22 649 581 529 529
Total cold run time: 51782 ms
Total hot run time: 49500 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209105561
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
+ }
+ Set<Slot> inputSlots = expression.getInputSlots();
+ ImmutableEqualSet<Slot> newEqualSet = equalAndConstantOptions.get().first;
+ Map<Slot, Literal> newConstants = equalAndConstantOptions.get().second;
+ // myInferConstantSlots : the slots that are inferred by this expression, not inferred by parent
+ // myInferConstantSlots[slot] = true means expression had contains conjunct `slot = constant`
+ Map<Slot, Boolean> myInferConstantSlots = Maps.newLinkedHashMapWithExpectedSize(
+ Math.max(0, newConstants.size() - parentConstants.size()));
+ for (Slot slot : newConstants.keySet()) {
+ if (!parentConstants.containsKey(slot)) {
+ myInferConstantSlots.put(slot, false);
+ }
+ }
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(conjunctions.size());
+ for (Expression child : conjunctions) {
+ Expression newChild = child;
+ // for expression, `a = 10 and a > b` will infer constant relation `a = 10`,
+ // need to replace a with 10 to this expression,
+ // for the first conjunction `a = 10`, no need to replace because after replace will get `10 = 10`,
+ // for the second conjunction `a > b`, need replace and got `10 > b`
+ if (needReplaceWithConstant(newChild, newConstants, myInferConstantSlots)) {
+ newChild = replaceConstants(newChild, useInnerInfer, context, newEqualSet, newConstants);
+ }
+ if (newChild.equals(BooleanLiteral.FALSE)) {
+ return BooleanLiteral.FALSE;
+ }
+ if (newChild instanceof And) {
+ builder.addAll(ExpressionUtils.extractConjunction(newChild));
+ } else {
+ builder.add(newChild);
+ }
+ }
+ // if the expression infer `slot = constant`, but not contains conjunct `slot = constant`, need to add it
+ for (Map.Entry<Slot, Boolean> entry : myInferConstantSlots.entrySet()) {
+ // if this expression don't contain the slot, no add it, to avoid the expression size increase too long
+ if (!entry.getValue() && inputSlots.contains(entry.getKey())) {
+ Slot slot = entry.getKey();
+ EqualTo equal = new EqualTo(slot, newConstants.get(slot), true);
+ builder.add(TypeCoercionUtils.processComparisonPredicate(equal));
+ }
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ // process OR expression
+ private Expression replaceOrConstants(Or expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> disjunctions = ExpressionUtils.extractDisjunction(expression);
+ ImmutableList.Builder<Expression> builder = ImmutableList.builderWithExpectedSize(disjunctions.size());
+ for (Expression child : disjunctions) {
+ Expression newChild = replaceConstants(child, useInnerInfer, context, parentEqualSet, parentConstants);
+ if (newChild.equals(BooleanLiteral.TRUE)) {
+ return BooleanLiteral.TRUE;
+ }
+ builder.add(newChild);
+ }
+ return expression.withChildren(builder.build());
+ }
+
+ private boolean needReplaceWithConstant(Expression expression, Map<Slot, Literal> constants,
+ Map<Slot, Boolean> myInferConstantSlots) {
+ if (expression instanceof EqualTo && expression.child(0) instanceof Slot) {
+ Slot slot = (Slot) expression.child(0);
+ if (myInferConstantSlots.get(slot) == Boolean.FALSE
+ && expression.child(1).equals(constants.get(slot))) {
+ myInferConstantSlots.put(slot, true);
+ return false;
+ }
+ }
+
+ return true;
+ }
+
+ private boolean canReplaceExpression(Expression expression) {
+ // 'a is not null', EliminateOuterJoin will call TypeUtils.isNotNull
+ if (ExpressionUtils.isGeneratedNotNull(expression)) {
+ return false;
+ }
+
+ // "https://doris.apache.org/docs/sql-manual/basic-element/operators/conditional-operators
+ // /full-text-search-operators", the match function require left is a slot, not a literal.
+ if (expression instanceof Match) {
+ return false;
+ }
+
+ // EliminateJoinByFK, join with materialize view need keep `a = b`.
+ // But for a common join, need eliminate `a = b`, after eliminate hash join equations,
+ // hash join will change to nested loop join.
+ // Join with materialize view will no more need `a = b` later.
+ // TODO: replace constants with `a = b`
+ if (expression instanceof EqualPredicate
+ && expression.children().stream().allMatch(SlotReference.class::isInstance)) {
+ SlotReference left = (SlotReference) expression.child(0);
+ SlotReference right = (SlotReference) expression.child(1);
+ return left.getQualifier().equals(right.getQualifier());
+ }
+
+ return true;
+ }
+
+ private Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> getChildEqualSetAndConstants(
+ LogicalPlan plan, ExpressionRewriteContext context) {
+ if (plan.children().size() == 1) {
+ DataTrait dataTrait = plan.child(0).getLogicalProperties().getTrait();
+ return Pair.of(dataTrait.getEqualSet(), getConstantUniforms(dataTrait.getAllUniformValues(), context));
+ } else {
+ Map<Slot, Literal> uniformConstants = Maps.newHashMap();
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>();
+ for (Plan child : plan.children()) {
+ uniformConstants.putAll(
+ getConstantUniforms(child.getLogicalProperties().getTrait().getAllUniformValues(), context));
+ newEqualSetBuilder.addEqualSet(child.getLogicalProperties().getTrait().getEqualSet());
+ }
+ return Pair.of(newEqualSetBuilder.build(), uniformConstants);
+ }
+ }
+
+ private Map<Slot, Literal> getConstantUniforms(Map<Slot, Optional<Expression>> uniformValues,
+ ExpressionRewriteContext context) {
+ Map<Slot, Literal> constantValues = Maps.newHashMap();
+ for (Map.Entry<Slot, Optional<Expression>> entry : uniformValues.entrySet()) {
+ Expression expr = entry.getValue().isPresent() ? entry.getValue().get() : null;
+ if (expr == null || !expr.isConstant()) {
+ continue;
+ }
+ if (!expr.isLiteral()) {
+ // uniforms values contains like 'cast(11 as smallint)'
+ expr = FoldConstantRule.evaluate(expr, context);
+ if (!expr.isLiteral()) {
+ continue;
+ }
+ }
+ constantValues.put(entry.getKey(), (Literal) expr);
+ }
+
+ return constantValues;
+ }
+
+ // Extract equal set and constants from conjunctions, then combine them with parentEqualSet and parentConstants.
+ // If met conflict constants relation, return optional.empty().
+ private Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> expandEqualSetAndConstants(
+ List<Expression> conjunctions,
+ boolean useInnerInfer,
+ ImmutableEqualSet<Slot> parentEqualSet,
+ Map<Slot, Literal> parentConstants) {
+ Map<Slot, Literal> newConstants = Maps.newLinkedHashMapWithExpectedSize(parentConstants.size());
+ newConstants.putAll(parentConstants);
+ ImmutableEqualSet.Builder<Slot> newEqualSetBuilder = new Builder<>(parentEqualSet);
+ for (Expression child : conjunctions) {
+ Optional<Pair<Slot, Expression>> equalItem = findValidEqualItem(child);
+ if (!equalItem.isPresent()) {
+ continue;
+ }
+ Slot slot = equalItem.get().first;
+ Expression expr = equalItem.get().second;
+ // for expression `a = 1 and a * 3 = 10`
+ // if it's in LogicalFilter, then we can infer constant relation `a = 1`, then have:
+ // `a = 1 and 3 * 3 = 10` => `a = 1 and FALSE` => `FALSE`
+ // but if it's in LogicalProject, then we shouldn't propagate `a=1` to this expression,
+ // because a may be null, then `a=1 and a * 3 = 10` should evaluate to `NULL` in the project.
+ if (!useInnerInfer && (slot.nullable() || expr.nullable())) {
+ continue;
+ }
+ if (expr instanceof Slot) {
+ newEqualSetBuilder.addEqualPair(slot, (Slot) expr);
+ } else if (!addConstant(newConstants, slot, (Literal) expr)) {
+ return Optional.empty();
+ }
+ }
+
+ ImmutableEqualSet<Slot> newEqualSet = newEqualSetBuilder.build();
+ List<Set<Slot>> multiEqualSlots = newEqualSet.calEqualSetList();
+ for (Set<Slot> slots : multiEqualSlots) {
+ Slot slot = slots.stream().filter(newConstants::containsKey).findFirst().orElse(null);
+ if (slot == null) {
+ continue;
+ }
+ Literal value = newConstants.get(slot);
+ for (Slot s : slots) {
+ if (!addConstant(newConstants, s, value)) {
+ return Optional.empty();
+ }
+ }
+ }
+
+ return Optional.of(Pair.of(newEqualSet, newConstants));
+ }
+
+ private boolean addConstant(Map<Slot, Literal> constants, Slot slot, Literal value) {
+ Literal existValue = constants.get(slot);
+ if (existValue == null) {
+ constants.put(slot, value);
+ return true;
+ }
+ return value.equals(existValue)
+ || value instanceof ComparableLiteral && existValue instanceof ComparableLiteral
+ && ((ComparableLiteral) value).compareTo((ComparableLiteral) existValue) == 0;
Review Comment:
fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209104843
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return filter.withConjunctsAndChild(newConjuncts, filter.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalHaving(LogicalHaving<? extends Plan> having, ExpressionRewriteContext context) {
+ having = visitChildren(this, having, context);
+ Expression oldPredicate = having.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(having, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return having;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
+ return having.withConjunctsAndChild(newConjuncts, having.child());
+ }
+ }
+
+ @Override
+ public Plan visitLogicalProject(LogicalProject<? extends Plan> project, ExpressionRewriteContext context) {
+ project = visitChildren(this, project, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(project, context);
+ List<NamedExpression> newProjects = project.getProjects().stream()
+ .map(expr -> replaceNameExpressionConstants(
+ expr, context, childEqualTrait.first, childEqualTrait.second))
+ .collect(ImmutableList.toImmutableList());
+ return newProjects.equals(project.getProjects()) ? project : project.withProjects(newProjects);
+ }
+
+ @Override
+ public Plan visitLogicalSort(LogicalSort<? extends Plan> sort, ExpressionRewriteContext context) {
+ sort = visitChildren(this, sort, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(sort, context);
+ // for be, order key must be a column, not a literal, so `order by 100#xx` is ok,
+ // but `order by 100` will make be core.
+ // so after replaced, we need to remove the constant expr.
+ List<OrderKey> newOrderKeys = sort.getOrderKeys().stream()
+ .map(key -> key.withExpression(
+ replaceConstants(key.getExpr(), false, context, childEqualTrait.first, childEqualTrait.second)))
+ .filter(key -> !key.getExpr().isConstant())
+ .collect(ImmutableList.toImmutableList());
+ if (newOrderKeys.isEmpty()) {
+ return sort.child();
+ } else if (!newOrderKeys.equals(sort.getOrderKeys())) {
+ return sort.withOrderKeys(newOrderKeys);
+ } else {
+ return sort;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalAggregate(LogicalAggregate<? extends Plan> aggregate, ExpressionRewriteContext context) {
+ aggregate = visitChildren(this, aggregate, context);
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait =
+ getChildEqualSetAndConstants(aggregate, context);
+
+ List<Expression> oldGroupByExprs = aggregate.getGroupByExpressions();
+ List<Expression> newGroupByExprs = Lists.newArrayListWithExpectedSize(oldGroupByExprs.size());
+ for (Expression expr : oldGroupByExprs) {
+ Expression newExpr = replaceConstants(expr, false, context, childEqualTrait.first, childEqualTrait.second);
+ if (!newExpr.isConstant()) {
+ newGroupByExprs.add(newExpr);
+ }
+ }
+
+ // group by with literal and empty group by are different.
+ // the former can return 0 row, the latter return at least 1 row.
+ // when met all group by expression are constant,
+ // 'eliminateGroupByConstant' will put a project(alias constant as slot) below the agg,
+ // but this rule cann't put a project below the agg, otherwise this rule may cause a dead loop,
+ // so when all replaced group by expression are constant, just let new group by add an origin group by.
+ if (newGroupByExprs.isEmpty() && !oldGroupByExprs.isEmpty()) {
+ newGroupByExprs.add(oldGroupByExprs.iterator().next());
+ }
+ Set<Expression> newGroupByExprSet = Sets.newHashSet(newGroupByExprs);
+
+ List<NamedExpression> oldOutputExprs = aggregate.getOutputExpressions();
+ List<NamedExpression> newOutputExprs = Lists.newArrayListWithExpectedSize(oldOutputExprs.size());
+ ImmutableList.Builder<NamedExpression> projectBuilder
+ = ImmutableList.builderWithExpectedSize(oldOutputExprs.size());
+
+ boolean containsConstantOutput = false;
+
+ // after normal agg, group by expressions and output expressions are slots,
+ // after this rule, they may rewrite to literal, since literal are not slot,
+ // we need eliminate the rewritten literals.
+ for (NamedExpression expr : oldOutputExprs) {
+ // ColumnPruning will also add all group by expression into output expressions
+ // agg output need contains group by expression
+ Expression replacedExpr = replaceConstants(expr, false, context,
+ childEqualTrait.first, childEqualTrait.second);
+ Expression newOutputExpr = newGroupByExprSet.contains(expr) ? expr : replacedExpr;
+ if (newOutputExpr instanceof NamedExpression) {
+ newOutputExprs.add((NamedExpression) newOutputExpr);
+ }
+
+ if (replacedExpr.isConstant()) {
+ projectBuilder.add(new Alias(expr.getExprId(), replacedExpr, expr.getName()));
+ containsConstantOutput = true;
+ } else {
+ Preconditions.checkArgument(newOutputExpr instanceof NamedExpression, newOutputExpr);
+ projectBuilder.add(((NamedExpression) newOutputExpr).toSlot());
+ }
+ }
+
+ if (newGroupByExprs.equals(oldGroupByExprs) && newOutputExprs.equals(oldOutputExprs)) {
+ return aggregate;
+ }
+
+ aggregate = aggregate.withGroupByAndOutput(newGroupByExprs, newOutputExprs);
+ if (containsConstantOutput) {
+ return PlanUtils.projectOrSelf(projectBuilder.build(), aggregate);
+ } else {
+ return aggregate;
+ }
+ }
+
+ @Override
+ public Plan visitLogicalRepeat(LogicalRepeat<? extends Plan> repeat, ExpressionRewriteContext context) {
+ repeat = visitChildren(this, repeat, context);
+ // TODO: process the repeat like agg ?
+ return repeat;
+ }
+
+ @Override
+ public Plan visitLogicalJoin(LogicalJoin<? extends Plan, ? extends Plan> join, ExpressionRewriteContext context) {
+ join = visitChildren(this, join, context);
+ // TODO: need rewrite the mark conditions ?
+ List<Expression> allJoinConjuncts = Stream.concat(join.getHashJoinConjuncts().stream(),
+ join.getOtherJoinConjuncts().stream())
+ .collect(ImmutableList.toImmutableList());
+ Expression oldPredicate = ExpressionUtils.and(allJoinConjuncts);
+ Expression newPredicate = replaceConstantsAndRewriteExpr(join, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return join;
+ }
+
+ // TODO: code from FindHashConditionForJoin
+ Pair<List<Expression>, List<Expression>> pair = JoinUtils.extractExpressionForHashTable(
+ join.left().getOutput(), join.right().getOutput(), ExpressionUtils.extractConjunction(newPredicate));
+
+ List<Expression> newHashJoinConjuncts = pair.first;
+ List<Expression> newOtherJoinConjuncts = pair.second;
+ JoinType joinType = join.getJoinType();
+ if (joinType == JoinType.CROSS_JOIN && !newHashJoinConjuncts.isEmpty()) {
+ joinType = JoinType.INNER_JOIN;
+ }
+
+ if (newHashJoinConjuncts.equals(join.getHashJoinConjuncts())
+ && newOtherJoinConjuncts.equals(join.getOtherJoinConjuncts())) {
+ return join;
+ }
+
+ return new LogicalJoin<>(joinType,
+ newHashJoinConjuncts,
+ newOtherJoinConjuncts,
+ join.getMarkJoinConjuncts(),
+ join.getDistributeHint(),
+ join.getMarkJoinSlotReference(),
+ join.children(), join.getJoinReorderContext());
+ }
+
+ // for sql: create table t as select cast('1' as varchar(30))
+ // the select will add a parent plan: result sink. the result sink contains a output slot reference, and its
+ // data type is varchar(30), but if replace the slot reference with a varchar literal '1', then the data type info
+ // varchar(30) will lost, because varchar literal '1' data type is always varchar(1), so t's column will get
+ // a error type.
+ // so we don't rewrite logical sink then.
+ // @Override
+ // public Plan visitLogicalSink(LogicalSink<? extends Plan> sink, ExpressionRewriteContext context) {
+ // sink = visitChildren(this, sink, context);
+ // Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait
+ // = getChildEqualSetAndConstants(sink, context);
+ // List<NamedExpression> newOutputExprs = sink.getOutputExprs().stream()
+ // .map(expr ->
+ // replaceNameExpressionConstants(expr, context, childEqualTrait.first, childEqualTrait.second))
+ // .collect(ImmutableList.toImmutableList());
+ // return newOutputExprs.equals(sink.getOutputExprs()) ? sink : sink.withOutputExprs(newOutputExprs);
+ // }
+
+ /**
+ * replace constants and rewrite expression.
+ */
+ @VisibleForTesting
+ public Expression replaceConstantsAndRewriteExpr(LogicalPlan plan, Expression expression,
+ boolean useInnerInfer, ExpressionRewriteContext context) {
+ // for expression `a = 1 and a + b = 2 and b + c = 2 and c + d =2 and ...`:
+ // propagate constant `a = 1`, then get `1 + b = 2`, after rewrite this expression, will get `b = 1`;
+ // then propagate constant `b = 1`, then get `1 + c = 2`, after rewrite this expression, will get `c = 1`,
+ // ...
+ // so constant propagate and rewrite expression need to do in a loop.
+ Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>> childEqualTrait = getChildEqualSetAndConstants(plan, context);
+ Expression afterExpression = expression;
+ for (int i = 0; i < 100; i++) {
+ Expression beforeExpression = afterExpression;
+ afterExpression = replaceConstants(beforeExpression, useInnerInfer, context,
+ childEqualTrait.first, childEqualTrait.second);
+ if (isExprEqualIgnoreOrder(beforeExpression, afterExpression)) {
+ break;
+ }
+ if (afterExpression.isLiteral()) {
+ break;
+ }
+ beforeExpression = afterExpression;
+ afterExpression = exprNormalAndOpt.rewrite(beforeExpression, context);
+ }
+ return afterExpression;
+ }
+
+ // process NameExpression
+ private NamedExpression replaceNameExpressionConstants(NamedExpression expr, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> equalSet, Map<Slot, Literal> constants) {
+
+ // if a project item is a slot reference, and the slot equals to a constant value, don't rewrite it.
+ // because rule `EliminateUnnecessaryProject ` can eliminate a project when the project's output slots equal to
+ // its child's output slots.
+ // for example, for `sink -> ... -> project(a, b, c) -> filter(a = 10)`
+ // if rewrite project to project(alias 10 as a, b, c), later other rule may prune `alias 10 as a`, and project
+ // will become project(b, c), so project and filter's output slot will not equal,
+ // then the project cannot be eliminated.
+ // so we don't replace SlotReference.
+ // for safety reason, we only replace Alias
+ if (!(expr instanceof Alias)) {
+ return expr;
+ }
+
+ // PushProjectThroughUnion require projection is a slot reference, or like (cast slot reference as xx);
+ // TODO: if PushProjectThroughUnion support projection like `literal as xx`, then delete this check.
+ if (ExpressionUtils.getExpressionCoveredByCast(expr.child(0)) instanceof SlotReference) {
+ return expr;
+ }
+
+ Expression newExpr = replaceConstants(expr, false, context, equalSet, constants);
+ if (newExpr instanceof NamedExpression) {
+ return (NamedExpression) newExpr;
+ } else {
+ return new Alias(expr.getExprId(), newExpr, expr.getName());
+ }
+ }
+
+ private Expression replaceConstants(Expression expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ if (expression instanceof And) {
+ return replaceAndConstants((And) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (expression instanceof Or) {
+ return replaceOrConstants((Or) expression, useInnerInfer, context, parentEqualSet, parentConstants);
+ } else if (!parentConstants.isEmpty()
+ && expression.anyMatch(e -> e instanceof Slot && parentConstants.containsKey(e))) {
+ Expression newExpr = ExpressionUtils.replaceIf(expression, parentConstants, this::canReplaceExpression);
+ if (!newExpr.equals(expression)) {
+ newExpr = FoldConstantRule.evaluate(newExpr, context);
+ }
+ return newExpr;
+ } else {
+ return expression;
+ }
+ }
+
+ // process AND expression
+ private Expression replaceAndConstants(And expression, boolean useInnerInfer, ExpressionRewriteContext context,
+ ImmutableEqualSet<Slot> parentEqualSet, Map<Slot, Literal> parentConstants) {
+ List<Expression> conjunctions = ExpressionUtils.extractConjunction(expression);
+ Optional<Pair<ImmutableEqualSet<Slot>, Map<Slot, Literal>>> equalAndConstantOptions =
+ expandEqualSetAndConstants(conjunctions, useInnerInfer, parentEqualSet, parentConstants);
+ // infer conflict constants like a = 10 and a = 30
+ if (!equalAndConstantOptions.isPresent()) {
+ return BooleanLiteral.FALSE;
Review Comment:
had consider a is nullable, the rewrite is safe. see annotation from function expandEqualSetAndConstants
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209093752
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ConstantPropagation.java:
##########
@@ -0,0 +1,601 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.properties.DataTrait;
+import org.apache.doris.nereids.properties.OrderKey;
+import org.apache.doris.nereids.rules.expression.ExpressionNormalizationAndOptimization;
+import org.apache.doris.nereids.rules.expression.ExpressionRewriteContext;
+import org.apache.doris.nereids.rules.expression.rules.FoldConstantRule;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.And;
+import org.apache.doris.nereids.trees.expressions.EqualPredicate;
+import org.apache.doris.nereids.trees.expressions.EqualTo;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Match;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.ComparableLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.JoinType;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalHaving;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalRepeat;
+import org.apache.doris.nereids.trees.plans.logical.LogicalSort;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.ImmutableEqualSet;
+import org.apache.doris.nereids.util.ImmutableEqualSet.Builder;
+import org.apache.doris.nereids.util.JoinUtils;
+import org.apache.doris.nereids.util.PlanUtils;
+import org.apache.doris.nereids.util.PredicateInferUtils;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.annotations.VisibleForTesting;
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Maps;
+import com.google.common.collect.Sets;
+import org.apache.hadoop.util.Lists;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Stream;
+
+/**
+ * constant propagation, like: a = 10 and a + b > 30 => a = 10 and 10 + b > 30,
+ * when processing a plan, it will collect all its children's equal sets and constants uniforms,
+ * then use them and the plan's expressions to infer more equal sets and constants uniforms,
+ * finally use the combine uniforms to replace this plan's expression's slot with literals.
+ */
+public class ConstantPropagation extends DefaultPlanRewriter<ExpressionRewriteContext> implements CustomRewriter {
+
+ private final ExpressionNormalizationAndOptimization exprNormalAndOpt
+ = new ExpressionNormalizationAndOptimization(false);
+
+ @Override
+ public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+ // logical apply uniform maybe not correct.
+ if (plan.anyMatch(LogicalApply.class::isInstance)) {
+ return plan;
+ }
+ ExpressionRewriteContext context = new ExpressionRewriteContext(jobContext.getCascadesContext());
+ return plan.accept(this, context);
+ }
+
+ @Override
+ public Plan visitLogicalFilter(LogicalFilter<? extends Plan> filter, ExpressionRewriteContext context) {
+ filter = visitChildren(this, filter, context);
+ Expression oldPredicate = filter.getPredicate();
+ Expression newPredicate = replaceConstantsAndRewriteExpr(filter, oldPredicate, true, context);
+ if (isExprEqualIgnoreOrder(oldPredicate, newPredicate)) {
+ return filter;
+ } else {
+ Set<Expression> newConjuncts = Sets.newLinkedHashSet(ExpressionUtils.extractConjunction(newPredicate));
Review Comment:
update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3079131960
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2952741706
<details>
<summary>ClickBench: <b>Total hot run time: 29.07 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9284067656c48e175b43cffa6577d6dfaec077b7, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.10
query3 0.26 0.20 0.19
query4 1.60 0.18 0.11
query5 0.45 0.42 0.43
query6 1.15 0.66 0.66
query7 0.03 0.02 0.02
query8 0.05 0.04 0.03
query9 0.60 0.50 0.51
query10 0.56 0.58 0.56
query11 0.15 0.11 0.11
query12 0.15 0.11 0.11
query13 0.62 0.59 0.59
query14 0.79 0.80 0.81
query15 0.88 0.87 0.85
query16 0.38 0.38 0.38
query17 1.05 1.01 1.00
query18 0.22 0.21 0.22
query19 1.91 1.88 1.84
query20 0.01 0.02 0.01
query21 15.40 0.92 0.55
query22 0.74 1.11 0.97
query23 14.73 1.38 0.66
query24 7.47 1.07 0.32
query25 0.28 0.12 0.20
query26 0.71 0.16 0.14
query27 0.06 0.06 0.05
query28 8.80 0.91 0.45
query29 12.65 4.13 3.47
query30 0.25 0.09 0.06
query31 2.84 0.62 0.40
query32 3.23 0.55 0.46
query33 3.08 3.09 3.15
query34 15.84 5.10 4.54
query35 4.46 4.55 4.48
query36 0.68 0.49 0.48
query37 0.09 0.06 0.06
query38 0.06 0.04 0.03
query39 0.03 0.03 0.02
query40 0.18 0.13 0.13
query41 0.08 0.02 0.02
query42 0.04 0.03 0.02
query43 0.04 0.04 0.03
Total cold run time: 102.77 s
Total hot run time: 29.07 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2953956857
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2875929857
<details>
<summary>TPC-H: <b>Total hot run time: 33739 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d39a11d88dea7be0865d4d4061a50ab3d8b34a69, data reload: false
------ Round 1 ----------------------------------
q1 25984 5032 4959 4959
q2 2071 299 176 176
q3 10384 1249 687 687
q4 10222 993 519 519
q5 7514 2383 2325 2325
q6 175 161 130 130
q7 909 757 608 608
q8 9319 1298 1055 1055
q9 6768 5049 5042 5042
q10 6856 2292 1894 1894
q11 475 290 267 267
q12 343 344 208 208
q13 17765 3661 3082 3082
q14 225 221 203 203
q15 520 487 498 487
q16 415 430 393 393
q17 618 847 366 366
q18 7655 7238 7177 7177
q19 1511 956 550 550
q20 322 327 218 218
q21 3830 3360 2438 2438
q22 1061 1005 955 955
Total cold run time: 114942 ms
Total hot run time: 33739 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5060 5028 5033 5028
q2 235 320 228 228
q3 2159 2626 2307 2307
q4 1355 1758 1367 1367
q5 4387 4367 4363 4363
q6 211 162 125 125
q7 2004 1949 1743 1743
q8 2536 2529 2489 2489
q9 7178 7169 6981 6981
q10 3051 3193 2764 2764
q11 574 517 497 497
q12 707 762 618 618
q13 3516 3868 3264 3264
q14 278 299 277 277
q15 539 480 490 480
q16 446 486 435 435
q17 1139 1546 1361 1361
q18 7807 7551 7385 7385
q19 799 811 923 811
q20 1985 2072 1833 1833
q21 5079 4660 4690 4660
q22 1098 1088 1048 1048
Total cold run time: 52143 ms
Total hot run time: 50064 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3027211509
<details>
<summary>ClickBench: <b>Total hot run time: 29.15 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce8c338875395acc46f8f4db2550a6b4de478f44, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.05
query3 0.24 0.08 0.07
query4 1.62 0.11 0.11
query5 0.43 0.43 0.41
query6 1.17 0.66 0.64
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.59 0.51 0.51
query10 0.57 0.58 0.57
query11 0.15 0.12 0.11
query12 0.15 0.12 0.11
query13 0.63 0.61 0.60
query14 0.82 0.81 0.81
query15 0.90 0.86 0.87
query16 0.39 0.37 0.39
query17 1.03 1.08 1.10
query18 0.22 0.21 0.21
query19 1.89 1.89 1.85
query20 0.01 0.01 0.02
query21 15.40 0.89 0.56
query22 0.76 1.25 0.63
query23 14.94 1.38 0.60
query24 6.93 1.35 0.36
query25 0.55 0.14 0.11
query26 0.67 0.17 0.13
query27 0.06 0.06 0.05
query28 9.60 0.95 0.47
query29 12.54 4.04 3.35
query30 0.25 0.09 0.07
query31 2.85 0.59 0.38
query32 3.25 0.55 0.46
query33 3.04 3.07 3.15
query34 16.15 5.44 4.79
query35 4.84 4.84 4.84
query36 0.70 0.51 0.49
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.03
query40 0.18 0.15 0.14
query41 0.08 0.03 0.03
query42 0.04 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 104.04 s
Total hot run time: 29.15 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3027297352
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2809415568
<details>
<summary>ClickBench: <b>Total hot run time: 29.12 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eaef1200d54db43a8b5ff63aa1975bd8d9b89cfb, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.12
query3 0.25 0.20 0.19
query4 1.60 0.19 0.20
query5 0.58 0.59 0.59
query6 1.19 0.73 0.72
query7 0.02 0.02 0.02
query8 0.04 0.04 0.04
query9 0.57 0.52 0.51
query10 0.57 0.57 0.56
query11 0.15 0.11 0.11
query12 0.14 0.11 0.12
query13 0.60 0.60 0.61
query14 1.15 1.20 1.22
query15 0.87 0.86 0.88
query16 0.39 0.38 0.38
query17 0.98 1.04 1.01
query18 0.21 0.19 0.20
query19 1.91 1.82 1.76
query20 0.02 0.01 0.02
query21 15.42 0.88 0.55
query22 0.76 1.14 0.70
query23 14.92 1.42 0.65
query24 7.42 1.60 0.33
query25 0.30 0.20 0.08
query26 0.66 0.16 0.13
query27 0.05 0.05 0.04
query28 8.64 0.89 0.43
query29 12.57 3.97 3.30
query30 0.25 0.09 0.07
query31 2.83 0.57 0.40
query32 3.23 0.54 0.46
query33 3.00 3.00 3.13
query34 15.79 5.07 4.50
query35 4.50 4.56 4.50
query36 0.68 0.51 0.48
query37 0.09 0.06 0.07
query38 0.05 0.05 0.04
query39 0.03 0.03 0.02
query40 0.17 0.15 0.13
query41 0.09 0.03 0.02
query42 0.04 0.02 0.03
query43 0.04 0.03 0.02
Total cold run time: 102.94 s
Total hot run time: 29.12 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2808270520
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2824605480
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2824741355
<details>
<summary>ClickBench: <b>Total hot run time: 29.65 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12489c4b0c61b9e294caec8c15a297f2c7171fbc, data reload: false
query1 0.03 0.03 0.03
query2 0.12 0.11 0.11
query3 0.26 0.20 0.19
query4 1.60 0.20 0.12
query5 0.56 0.55 0.56
query6 1.20 0.72 0.72
query7 0.02 0.02 0.01
query8 0.04 0.04 0.04
query9 0.58 0.50 0.51
query10 0.58 0.57 0.58
query11 0.15 0.11 0.11
query12 0.16 0.12 0.11
query13 0.61 0.60 0.59
query14 1.16 1.16 1.19
query15 0.87 0.84 0.86
query16 0.38 0.39 0.38
query17 1.01 1.03 1.03
query18 0.21 0.20 0.20
query19 1.87 1.78 1.76
query20 0.01 0.01 0.01
query21 15.40 0.89 0.54
query22 0.77 1.13 0.70
query23 14.90 1.39 0.64
query24 7.40 1.22 0.96
query25 0.48 0.26 0.05
query26 0.56 0.16 0.15
query27 0.06 0.05 0.06
query28 9.43 0.87 0.44
query29 12.65 3.98 3.33
query30 0.25 0.10 0.06
query31 2.82 0.60 0.38
query32 3.22 0.54 0.48
query33 3.00 2.99 3.06
query34 15.78 5.15 4.53
query35 4.54 4.54 4.55
query36 0.66 0.51 0.48
query37 0.09 0.06 0.06
query38 0.06 0.04 0.03
query39 0.03 0.03 0.02
query40 0.17 0.13 0.12
query41 0.08 0.03 0.02
query42 0.03 0.02 0.02
query43 0.03 0.04 0.03
Total cold run time: 103.83 s
Total hot run time: 29.65 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2822973051
<details>
<summary>TPC-DS: <b>Total hot run time: 192813 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51d565f4fc52c0b7e4d1370e096c79be06110bb0, data reload: false
query1 1443 1087 1059 1059
query2 6060 1774 1767 1767
query3 11003 4704 4564 4564
query4 26008 23575 23420 23420
query5 4406 615 451 451
query6 304 207 201 201
query7 3991 499 280 280
query8 298 256 242 242
query9 8486 2554 2563 2554
query10 477 349 291 291
query11 15354 15092 14816 14816
query12 156 110 107 107
query13 1578 530 405 405
query14 8911 6115 6092 6092
query15 201 186 172 172
query16 7675 591 458 458
query17 1187 737 605 605
query18 2022 415 314 314
query19 193 196 169 169
query20 135 119 127 119
query21 213 126 111 111
query22 4643 4681 4603 4603
query23 34698 33866 33596 33596
query24 8916 2450 2429 2429
query25 513 468 396 396
query26 1257 282 154 154
query27 2832 513 342 342
query28 4620 2139 2146 2139
query29 712 579 439 439
query30 287 224 199 199
query31 904 867 777 777
query32 73 62 94 62
query33 544 372 307 307
query34 802 877 522 522
query35 838 848 762 762
query36 983 992 905 905
query37 118 103 79 79
query38 4344 4226 4239 4226
query39 1504 1455 1628 1455
query40 215 123 117 117
query41 53 52 52 52
query42 116 120 110 110
query43 503 501 466 466
query44 1288 812 828 812
query45 183 177 168 168
query46 838 1027 642 642
query47 1913 1905 1803 1803
query48 388 423 315 315
query49 838 506 435 435
query50 653 692 402 402
query51 4239 4208 4156 4156
query52 108 109 100 100
query53 239 256 191 191
query54 598 577 505 505
query55 82 85 79 79
query56 307 333 292 292
query57 1165 1208 1152 1152
query58 281 269 274 269
query59 2677 2852 2711 2711
query60 322 321 305 305
query61 133 130 122 122
query62 773 751 676 676
query63 226 187 195 187
query64 4038 1046 686 686
query65 4449 4396 4397 4396
query66 1009 403 307 307
query67 15727 15553 15683 15553
query68 8065 883 530 530
query69 502 304 273 273
query70 1175 1149 1077 1077
query71 422 326 317 317
query72 5735 5021 4885 4885
query73 702 671 345 345
query74 8861 9113 9029 9029
query75 3193 3188 2708 2708
query76 3152 1183 761 761
query77 488 383 282 282
query78 9039 9094 8477 8477
query79 1883 865 566 566
query80 644 512 449 449
query81 493 246 218 218
query82 198 123 95 95
query83 251 249 248 248
query84 253 107 82 82
query85 705 304 256 256
query86 364 306 292 292
query87 4447 4432 4391 4391
query88 3350 2233 2221 2221
query89 398 316 278 278
query90 1976 210 211 210
query91 136 139 109 109
query92 69 57 57 57
query93 2262 940 574 574
query94 676 417 306 306
query95 375 303 287 287
query96 481 565 277 277
query97 3092 3279 3132 3132
query98 218 203 202 202
query99 1388 1425 1281 1281
Total cold run time: 277043 ms
Total hot run time: 192813 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2829960114
<details>
<summary>TPC-H: <b>Total hot run time: 34644 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c93fc833a1a8da818b939c19fbf70373fb15231, data reload: false
------ Round 1 ----------------------------------
q1 26138 5067 5004 5004
q2 2063 271 188 188
q3 10392 1243 703 703
q4 10228 985 543 543
q5 7529 2347 2356 2347
q6 184 172 135 135
q7 916 742 620 620
q8 9299 1293 1078 1078
q9 6776 5078 5120 5078
q10 6796 2322 1875 1875
q11 470 285 265 265
q12 350 346 222 222
q13 17894 3733 3176 3176
q14 229 229 218 218
q15 523 486 485 485
q16 452 452 400 400
q17 606 860 360 360
q18 7497 7110 7159 7110
q19 1216 955 542 542
q20 354 330 224 224
q21 4294 3295 3115 3115
q22 1090 1049 956 956
Total cold run time: 115296 ms
Total hot run time: 34644 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5116 5056 5065 5056
q2 239 334 228 228
q3 2173 2663 2255 2255
q4 1383 1786 1388 1388
q5 4453 4412 4436 4412
q6 216 162 129 129
q7 2015 1923 1762 1762
q8 2568 2602 2511 2511
q9 7307 7224 7265 7224
q10 2975 3160 2790 2790
q11 579 513 506 506
q12 714 760 618 618
q13 3527 3907 3323 3323
q14 280 299 283 283
q15 542 487 483 483
q16 479 530 481 481
q17 1144 1467 1407 1407
q18 7744 7541 7521 7521
q19 830 824 973 824
q20 1908 2016 1855 1855
q21 5572 4829 4766 4766
q22 1113 1127 1040 1040
Total cold run time: 52877 ms
Total hot run time: 50862 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833916590
<details>
<summary>ClickBench: <b>Total hot run time: 29.65 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 125afa5d3078f0fc800b481e5d33c85e0970562e, data reload: false
query1 0.04 0.04 0.04
query2 0.12 0.10 0.12
query3 0.24 0.19 0.19
query4 1.60 0.19 0.19
query5 0.58 0.60 0.61
query6 1.19 0.71 0.71
query7 0.02 0.02 0.02
query8 0.05 0.03 0.04
query9 0.58 0.52 0.51
query10 0.59 0.58 0.56
query11 0.15 0.11 0.11
query12 0.14 0.12 0.13
query13 0.60 0.60 0.60
query14 1.14 1.20 1.20
query15 0.89 0.85 0.84
query16 0.37 0.39 0.40
query17 1.04 1.05 1.08
query18 0.21 0.20 0.20
query19 1.87 1.77 1.86
query20 0.01 0.01 0.01
query21 15.40 0.88 0.55
query22 0.75 1.23 0.75
query23 14.80 1.37 0.63
query24 7.78 0.89 0.63
query25 0.49 0.19 0.12
query26 0.63 0.17 0.13
query27 0.05 0.05 0.05
query28 8.94 0.88 0.43
query29 12.53 3.91 3.29
query30 0.25 0.09 0.06
query31 2.83 0.60 0.38
query32 3.23 0.56 0.47
query33 3.08 3.09 3.05
query34 15.88 5.12 4.48
query35 4.55 4.57 4.58
query36 0.66 0.50 0.49
query37 0.09 0.07 0.07
query38 0.06 0.04 0.04
query39 0.03 0.03 0.02
query40 0.18 0.14 0.14
query41 0.07 0.03 0.02
query42 0.03 0.03 0.02
query43 0.04 0.03 0.02
Total cold run time: 103.78 s
Total hot run time: 29.65 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2837747176
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833370640
<details>
<summary>TPC-H: <b>Total hot run time: 35179 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 484c6bfb7960914a92b656959003939aa69b130c, data reload: false
------ Round 1 ----------------------------------
q1 26647 5130 5093 5093
q2 2081 289 191 191
q3 10388 1265 706 706
q4 10225 995 555 555
q5 7483 2430 2404 2404
q6 191 168 135 135
q7 903 737 637 637
q8 9327 1340 1158 1158
q9 6761 5069 5081 5069
q10 6814 2313 1887 1887
q11 473 296 278 278
q12 354 356 219 219
q13 17773 3687 3157 3157
q14 238 227 221 221
q15 542 493 483 483
q16 461 454 404 404
q17 604 863 379 379
q18 7520 7193 7125 7125
q19 1213 938 590 590
q20 342 340 226 226
q21 4525 3602 3312 3312
q22 1064 1001 950 950
Total cold run time: 115929 ms
Total hot run time: 35179 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5176 5143 5108 5108
q2 247 330 234 234
q3 2114 2641 2297 2297
q4 1386 1899 1536 1536
q5 4591 4453 4362 4362
q6 213 169 128 128
q7 1982 1904 1694 1694
q8 2622 2618 2505 2505
q9 7110 7205 7093 7093
q10 3040 3153 2738 2738
q11 591 508 485 485
q12 680 772 608 608
q13 3552 4021 3316 3316
q14 299 303 271 271
q15 517 474 468 468
q16 461 501 464 464
q17 1161 1589 1363 1363
q18 7718 7550 7327 7327
q19 852 825 859 825
q20 2013 2033 1841 1841
q21 5171 4682 4630 4630
q22 1081 1018 991 991
Total cold run time: 52577 ms
Total hot run time: 50284 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838151048
<details>
<summary>ClickBench: <b>Total hot run time: 29.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 135312542e23c5083a8d9116bbce26d9e4f6f1c0, data reload: false
query1 0.03 0.04 0.03
query2 0.12 0.10 0.12
query3 0.25 0.19 0.19
query4 1.59 0.19 0.19
query5 0.59 0.61 0.60
query6 1.19 0.73 0.73
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.57 0.52 0.52
query10 0.57 0.57 0.57
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.61 0.60 0.59
query14 0.78 0.80 0.80
query15 0.90 0.86 0.85
query16 0.41 0.40 0.38
query17 1.04 1.04 1.03
query18 0.22 0.20 0.20
query19 1.92 1.87 1.77
query20 0.02 0.01 0.02
query21 15.40 0.90 0.56
query22 0.76 1.21 0.67
query23 14.94 1.39 0.65
query24 6.80 1.36 0.90
query25 0.46 0.14 0.14
query26 0.55 0.16 0.15
query27 0.05 0.05 0.05
query28 10.25 0.83 0.42
query29 12.57 4.00 3.32
query30 0.25 0.10 0.06
query31 2.83 0.63 0.39
query32 3.22 0.54 0.48
query33 3.14 3.03 3.03
query34 15.74 5.11 4.48
query35 4.54 4.47 4.52
query36 0.66 0.49 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.03
query40 0.16 0.14 0.13
query41 0.09 0.03 0.03
query42 0.03 0.02 0.03
query43 0.03 0.03 0.03
Total cold run time: 103.83 s
Total hot run time: 29.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2833095630
<details>
<summary>TPC-DS: <b>Total hot run time: 184564 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit acef217490988ac1b4f5bb9e8a5496b5225ff8d3, data reload: false
query1 995 487 496 487
query2 6561 1875 1931 1875
query3 6736 224 230 224
query4 26037 23390 23092 23092
query5 4380 619 474 474
query6 302 205 196 196
query7 4617 482 280 280
query8 305 234 237 234
query9 8621 2550 2562 2550
query10 478 323 262 262
query11 15438 15405 14811 14811
query12 157 109 109 109
query13 1654 535 404 404
query14 8717 6083 6182 6083
query15 207 188 170 170
query16 7198 657 477 477
query17 1225 738 579 579
query18 1971 407 305 305
query19 194 197 164 164
query20 121 114 116 114
query21 213 125 115 115
query22 4325 4198 3917 3917
query23 34080 33069 33100 33069
query24 8508 2384 2376 2376
query25 576 478 427 427
query26 1244 278 216 216
query27 2688 510 330 330
query28 4343 2116 2100 2100
query29 736 536 422 422
query30 276 214 179 179
query31 911 791 721 721
query32 76 62 66 62
query33 548 352 324 324
query34 786 862 507 507
query35 789 816 721 721
query36 939 971 851 851
query37 110 101 76 76
query38 4253 4076 4147 4076
query39 1429 1383 1405 1383
query40 207 116 105 105
query41 57 55 53 53
query42 115 103 101 101
query43 517 511 491 491
query44 1281 803 787 787
query45 186 171 172 171
query46 844 1014 618 618
query47 1772 1814 1732 1732
query48 362 398 294 294
query49 803 526 428 428
query50 635 691 405 405
query51 4058 4077 4047 4047
query52 102 112 94 94
query53 218 254 185 185
query54 560 591 495 495
query55 82 79 79 79
query56 320 294 272 272
query57 1144 1149 1054 1054
query58 269 259 264 259
query59 2635 2790 2660 2660
query60 327 317 289 289
query61 122 121 126 121
query62 807 720 685 685
query63 213 183 180 180
query64 4347 996 658 658
query65 4319 4253 4197 4197
query66 1146 404 351 351
query67 15849 15263 15324 15263
query68 8377 882 518 518
query69 476 295 260 260
query70 1213 1079 1028 1028
query71 452 311 311 311
query72 5627 4647 4689 4647
query73 717 607 346 346
query74 9099 8993 8711 8711
query75 3962 3205 2735 2735
query76 3779 1186 746 746
query77 784 375 286 286
query78 9230 9307 8719 8719
query79 2234 873 619 619
query80 608 523 424 424
query81 461 264 223 223
query82 416 126 96 96
query83 288 257 236 236
query84 296 109 83 83
query85 738 321 274 274
query86 344 296 303 296
query87 4369 4357 4306 4306
query88 3249 2189 2177 2177
query89 391 315 282 282
query90 1923 203 210 203
query91 144 136 109 109
query92 72 59 58 58
query93 1239 958 587 587
query94 673 402 291 291
query95 373 282 279 279
query96 476 574 271 271
query97 3142 3224 3135 3135
query98 233 213 200 200
query99 1448 1404 1272 1272
Total cold run time: 272874 ms
Total hot run time: 184564 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838136254
<details>
<summary>TPC-DS: <b>Total hot run time: 191427 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 135312542e23c5083a8d9116bbce26d9e4f6f1c0, data reload: false
query1 1422 1087 1086 1086
query2 6140 1815 1819 1815
query3 11029 4522 4206 4206
query4 25606 23701 23397 23397
query5 4901 632 467 467
query6 322 204 200 200
query7 4004 491 278 278
query8 305 256 238 238
query9 8529 2592 2589 2589
query10 496 326 267 267
query11 15216 15049 15163 15049
query12 171 115 112 112
query13 1584 510 436 436
query14 9758 6349 6117 6117
query15 210 193 167 167
query16 7618 577 476 476
query17 1186 768 610 610
query18 2023 417 330 330
query19 203 206 170 170
query20 138 134 127 127
query21 210 126 117 117
query22 4586 4520 4301 4301
query23 34219 33899 33310 33310
query24 8376 2392 2417 2392
query25 484 453 395 395
query26 940 282 161 161
query27 2951 520 351 351
query28 4688 2102 2086 2086
query29 630 566 446 446
query30 277 230 202 202
query31 848 815 752 752
query32 74 75 63 63
query33 546 371 340 340
query34 787 867 530 530
query35 782 832 743 743
query36 959 988 897 897
query37 123 106 85 85
query38 4240 4175 4135 4135
query39 1478 1432 1489 1432
query40 217 135 109 109
query41 55 52 54 52
query42 130 114 112 112
query43 512 509 484 484
query44 1327 800 809 800
query45 185 178 169 169
query46 849 1030 639 639
query47 1840 1900 1786 1786
query48 370 421 302 302
query49 712 514 439 439
query50 659 715 418 418
query51 4181 4320 4187 4187
query52 116 114 99 99
query53 225 252 180 180
query54 579 586 515 515
query55 94 85 81 81
query56 329 276 286 276
query57 1160 1236 1133 1133
query58 281 258 249 249
query59 2886 2784 2705 2705
query60 333 331 301 301
query61 129 141 126 126
query62 749 737 686 686
query63 226 186 193 186
query64 3217 1049 709 709
query65 4415 4320 4415 4320
query66 787 408 309 309
query67 15975 15465 15344 15344
query68 9134 892 509 509
query69 478 300 262 262
query70 1221 1057 1082 1057
query71 478 320 302 302
query72 5280 4801 4743 4743
query73 723 611 343 343
query74 8818 9008 8813 8813
query75 4285 3210 2691 2691
query76 3712 1194 757 757
query77 801 376 280 280
query78 9260 9714 8693 8693
query79 1241 870 558 558
query80 646 518 454 454
query81 475 257 214 214
query82 429 123 95 95
query83 247 254 233 233
query84 245 107 96 96
query85 750 345 280 280
query86 342 307 306 306
query87 4365 4336 4233 4233
query88 2845 2227 2238 2227
query89 399 315 283 283
query90 1872 203 213 203
query91 145 145 112 112
query92 72 62 57 57
query93 1075 957 587 587
query94 658 405 315 315
query95 369 298 283 283
query96 486 563 273 273
query97 3145 3234 3113 3113
query98 233 214 209 209
query99 1571 1397 1255 1255
Total cold run time: 276397 ms
Total hot run time: 191427 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2838948310
<details>
<summary>TPC-DS: <b>Total hot run time: 192123 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ab930e4985c14df6decf4c7d449aea9ef0685c2, data reload: false
query1 1410 1108 1044 1044
query2 6225 1913 1867 1867
query3 11161 4379 4476 4379
query4 52816 25609 23379 23379
query5 4985 527 459 459
query6 332 194 203 194
query7 4871 500 282 282
query8 298 245 238 238
query9 5406 2570 2576 2570
query10 429 320 255 255
query11 14997 14980 14850 14850
query12 153 103 100 100
query13 1017 515 389 389
query14 10085 6378 6303 6303
query15 198 214 175 175
query16 7145 676 487 487
query17 1091 752 609 609
query18 1583 430 324 324
query19 205 200 174 174
query20 133 132 126 126
query21 211 132 107 107
query22 4401 4423 4305 4305
query23 34097 33531 33776 33531
query24 6650 2393 2443 2393
query25 475 483 404 404
query26 717 280 151 151
query27 2520 494 346 346
query28 3123 2104 2106 2104
query29 585 593 435 435
query30 274 228 193 193
query31 828 830 746 746
query32 74 65 69 65
query33 453 361 306 306
query34 766 840 530 530
query35 779 858 765 765
query36 970 981 875 875
query37 117 100 82 82
query38 4324 4274 4106 4106
query39 1507 1434 1402 1402
query40 208 119 106 106
query41 56 52 53 52
query42 121 115 107 107
query43 494 487 501 487
query44 1303 816 845 816
query45 190 177 171 171
query46 866 1033 647 647
query47 1873 1854 1792 1792
query48 384 426 306 306
query49 715 549 436 436
query50 659 721 424 424
query51 4171 4279 4152 4152
query52 112 110 98 98
query53 267 261 185 185
query54 588 581 504 504
query55 80 85 87 85
query56 310 295 294 294
query57 1212 1197 1144 1144
query58 280 267 253 253
query59 2855 3027 2830 2830
query60 322 329 306 306
query61 135 132 127 127
query62 732 758 668 668
query63 230 186 189 186
query64 1783 1011 727 727
query65 4318 4227 4200 4200
query66 711 393 301 301
query67 15967 15510 15366 15366
query68 7167 828 560 560
query69 535 297 270 270
query70 1193 1127 1099 1099
query71 501 304 294 294
query72 5916 4878 4992 4878
query73 1467 709 346 346
query74 9205 9123 8889 8889
query75 3888 3230 2647 2647
query76 4270 1178 744 744
query77 663 392 278 278
query78 9406 9328 8696 8696
query79 2234 799 561 561
query80 599 525 508 508
query81 485 264 220 220
query82 471 134 99 99
query83 403 259 223 223
query84 302 107 87 87
query85 758 319 278 278
query86 370 283 294 283
query87 4332 4389 4403 4389
query88 3711 2240 2193 2193
query89 411 322 273 273
query90 1819 230 218 218
query91 147 148 107 107
query92 70 70 55 55
query93 1771 967 596 596
query94 674 407 316 316
query95 370 296 295 295
query96 493 557 274 274
query97 3162 3204 3123 3123
query98 232 203 200 200
query99 1470 1382 1304 1304
Total cold run time: 296848 ms
Total hot run time: 192123 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3052952598
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3053081191
<details>
<summary>TPC-H: <b>Total hot run time: 33153 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ac0a854575f8810270790b6a58154b043b9529c, data reload: false
------ Round 1 ----------------------------------
q1 17605 5250 5020 5020
q2 1932 286 186 186
q3 10297 1275 737 737
q4 10248 1044 534 534
q5 8016 2319 2337 2319
q6 179 159 132 132
q7 911 741 622 622
q8 9328 1305 1066 1066
q9 6865 5093 5137 5093
q10 6881 2359 1969 1969
q11 501 292 283 283
q12 332 342 214 214
q13 17768 3736 3054 3054
q14 219 225 221 221
q15 568 491 480 480
q16 421 432 370 370
q17 590 863 364 364
q18 7449 7098 7087 7087
q19 1621 943 541 541
q20 341 340 226 226
q21 3713 3123 2336 2336
q22 350 328 299 299
Total cold run time: 106135 ms
Total hot run time: 33153 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5190 5118 5059 5059
q2 247 316 216 216
q3 2168 2706 2306 2306
q4 1357 1798 1331 1331
q5 4208 4292 4680 4292
q6 212 167 125 125
q7 2050 2020 1820 1820
q8 2609 2556 2544 2544
q9 7261 7415 7299 7299
q10 3083 3343 2922 2922
q11 584 524 492 492
q12 708 793 625 625
q13 3807 3964 3401 3401
q14 294 318 294 294
q15 517 495 480 480
q16 457 495 451 451
q17 1210 1486 1440 1440
q18 7773 7808 7687 7687
q19 854 945 1133 945
q20 2016 2056 1830 1830
q21 4796 4456 4309 4309
q22 652 602 538 538
Total cold run time: 52053 ms
Total hot run time: 50406 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2893790335
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2890659444
<details>
<summary>TPC-DS: <b>Total hot run time: 193660 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ee31a1ae3d3e603ceadb2c81ce12e1aec375f8da, data reload: false
query1 1439 1084 1061 1061
query2 6378 1786 1825 1786
query3 11022 4639 4418 4418
query4 54861 24789 23535 23535
query5 5164 569 471 471
query6 426 218 215 215
query7 4883 517 314 314
query8 326 271 248 248
query9 5349 2667 2655 2655
query10 439 356 271 271
query11 15295 15194 15598 15194
query12 165 117 106 106
query13 1057 549 432 432
query14 10187 6273 6269 6269
query15 216 215 192 192
query16 7181 670 511 511
query17 1133 742 607 607
query18 1633 420 322 322
query19 216 208 195 195
query20 134 142 124 124
query21 215 129 110 110
query22 4514 4521 4344 4344
query23 34311 33684 33690 33684
query24 6485 2375 2439 2375
query25 467 482 466 466
query26 699 274 164 164
query27 2253 527 359 359
query28 2963 2175 2170 2170
query29 600 581 443 443
query30 281 233 197 197
query31 858 824 742 742
query32 78 63 64 63
query33 456 396 323 323
query34 783 869 544 544
query35 809 836 751 751
query36 926 1008 916 916
query37 109 99 76 76
query38 4284 4344 4262 4262
query39 1547 1441 1493 1441
query40 212 139 124 124
query41 56 56 53 53
query42 129 111 110 110
query43 509 516 470 470
query44 1366 890 877 877
query45 187 185 174 174
query46 881 1057 684 684
query47 1887 1884 1780 1780
query48 423 455 323 323
query49 706 559 457 457
query50 666 708 429 429
query51 4178 4249 4181 4181
query52 112 111 104 104
query53 230 262 186 186
query54 601 580 527 527
query55 89 99 92 92
query56 323 329 331 329
query57 1199 1190 1144 1144
query58 273 278 266 266
query59 2682 2759 2694 2694
query60 373 317 323 317
query61 137 130 138 130
query62 754 721 675 675
query63 228 188 220 188
query64 1779 1021 686 686
query65 4325 4227 4200 4200
query66 714 400 311 311
query67 15835 15651 15487 15487
query68 6718 892 538 538
query69 560 319 278 278
query70 1203 1116 1110 1110
query71 510 326 299 299
query72 5926 4867 4957 4867
query73 1518 697 371 371
query74 8992 9186 8941 8941
query75 4016 3183 2687 2687
query76 4241 1195 757 757
query77 678 391 289 289
query78 9447 9416 8775 8775
query79 2016 882 614 614
query80 596 530 545 530
query81 471 258 223 223
query82 421 126 96 96
query83 251 249 234 234
query84 296 106 90 90
query85 758 318 279 279
query86 344 308 281 281
query87 4426 4503 4382 4382
query88 3410 2377 2353 2353
query89 410 328 276 276
query90 1878 205 204 204
query91 144 144 113 113
query92 69 68 57 57
query93 1185 953 591 591
query94 678 405 346 346
query95 368 290 290 290
query96 497 578 292 292
query97 2748 2816 2626 2626
query98 233 212 207 207
query99 1703 1405 1310 1310
Total cold run time: 297707 ms
Total hot run time: 193660 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2900637751
<details>
<summary>TPC-H: <b>Total hot run time: 34356 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd9a107d966f7614132b60ddf52bb5c27d8ae0ff, data reload: false
------ Round 1 ----------------------------------
q1 26202 5152 5131 5131
q2 2074 271 181 181
q3 10410 1251 735 735
q4 10367 1042 564 564
q5 7621 2408 2423 2408
q6 186 168 138 138
q7 933 763 626 626
q8 9343 1366 1155 1155
q9 7149 5155 5104 5104
q10 6886 2319 1893 1893
q11 493 293 278 278
q12 353 370 223 223
q13 17785 3708 3150 3150
q14 240 236 220 220
q15 548 499 490 490
q16 432 434 382 382
q17 634 876 381 381
q18 7463 7122 7179 7122
q19 1687 960 579 579
q20 351 334 216 216
q21 3917 2647 2423 2423
q22 1040 1041 957 957
Total cold run time: 116114 ms
Total hot run time: 34356 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5273 5212 5141 5141
q2 255 331 237 237
q3 2166 2656 2295 2295
q4 1338 1772 1425 1425
q5 4452 4354 4394 4354
q6 218 171 133 133
q7 2007 1961 1785 1785
q8 2612 2564 2500 2500
q9 7162 7199 6954 6954
q10 3042 3206 2772 2772
q11 596 522 522 522
q12 681 780 611 611
q13 3532 3921 3311 3311
q14 278 310 301 301
q15 543 486 477 477
q16 460 491 462 462
q17 1184 1564 1411 1411
q18 7708 7532 7349 7349
q19 862 898 922 898
q20 2004 2012 1867 1867
q21 4987 4501 4527 4501
q22 1103 1067 999 999
Total cold run time: 52463 ms
Total hot run time: 50305 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2943534750
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2944093807
<details>
<summary>ClickBench: <b>Total hot run time: 29.44 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c0c5557ef07d1fbe8f42072b38acc17bfaed12bd, data reload: false
query1 0.04 0.03 0.03
query2 0.12 0.11 0.10
query3 0.26 0.19 0.20
query4 1.59 0.20 0.19
query5 0.45 0.45 0.47
query6 1.18 0.67 0.67
query7 0.03 0.02 0.01
query8 0.04 0.04 0.03
query9 0.58 0.53 0.51
query10 0.59 0.60 0.57
query11 0.16 0.12 0.12
query12 0.15 0.12 0.12
query13 0.62 0.61 0.60
query14 0.80 0.82 0.82
query15 0.89 0.85 0.87
query16 0.39 0.38 0.37
query17 1.04 1.03 1.01
query18 0.22 0.21 0.22
query19 1.90 1.94 1.82
query20 0.02 0.01 0.01
query21 15.41 0.93 0.54
query22 0.74 1.18 0.77
query23 14.84 1.37 0.62
query24 6.90 0.89 1.51
query25 0.50 0.16 0.08
query26 0.66 0.16 0.14
query27 0.05 0.05 0.05
query28 10.13 0.96 0.47
query29 12.60 3.96 3.36
query30 0.26 0.10 0.06
query31 2.81 0.58 0.39
query32 3.23 0.56 0.46
query33 3.10 3.10 3.08
query34 15.72 5.22 4.50
query35 4.58 4.58 4.53
query36 0.65 0.50 0.49
query37 0.09 0.06 0.06
query38 0.06 0.04 0.04
query39 0.03 0.02 0.03
query40 0.18 0.14 0.14
query41 0.07 0.02 0.02
query42 0.04 0.02 0.02
query43 0.05 0.03 0.04
Total cold run time: 103.77 s
Total hot run time: 29.44 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2798434822
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2795790785
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2796694902
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [draft](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2796098356
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2952728100
<details>
<summary>TPC-H: <b>Total hot run time: 33828 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9284067656c48e175b43cffa6577d6dfaec077b7, data reload: false
------ Round 1 ----------------------------------
q1 26053 5007 5042 5007
q2 1970 279 185 185
q3 10376 1362 674 674
q4 10217 1034 538 538
q5 7662 2364 2350 2350
q6 182 164 134 134
q7 923 727 635 635
q8 9324 1321 1084 1084
q9 6751 5103 5111 5103
q10 6871 2330 1913 1913
q11 512 277 278 277
q12 352 349 216 216
q13 17780 3700 3099 3099
q14 235 230 213 213
q15 583 488 480 480
q16 421 438 386 386
q17 593 871 365 365
q18 7717 7316 7082 7082
q19 1984 960 563 563
q20 330 328 214 214
q21 3767 3156 2345 2345
q22 1039 1017 965 965
Total cold run time: 115642 ms
Total hot run time: 33828 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5280 5074 5107 5074
q2 247 327 222 222
q3 2168 2658 2279 2279
q4 1411 1776 1338 1338
q5 4483 4368 4432 4368
q6 221 171 128 128
q7 2040 1938 1789 1789
q8 2605 2587 2467 2467
q9 7176 7174 6934 6934
q10 3040 3224 2694 2694
q11 562 521 495 495
q12 688 765 636 636
q13 3467 3846 3288 3288
q14 287 310 277 277
q15 530 491 500 491
q16 453 491 450 450
q17 1147 1605 1333 1333
q18 7825 7628 7542 7542
q19 823 784 861 784
q20 2019 2070 1846 1846
q21 4811 4585 4472 4472
q22 1134 1080 1039 1039
Total cold run time: 52417 ms
Total hot run time: 49946 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-2914887744
<details>
<summary>TPC-H: <b>Total hot run time: 33782 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5de5b4e8038ca6b71c5470481e59e65c6de3bb9f, data reload: false
------ Round 1 ----------------------------------
q1 26258 5014 4976 4976
q2 2082 282 188 188
q3 10501 1267 690 690
q4 10243 1002 510 510
q5 7669 2395 2332 2332
q6 187 174 134 134
q7 916 714 608 608
q8 9303 1296 1089 1089
q9 6853 5074 5094 5074
q10 6866 2312 1871 1871
q11 498 286 282 282
q12 339 349 203 203
q13 17777 3653 3091 3091
q14 239 229 207 207
q15 530 482 494 482
q16 434 436 386 386
q17 616 863 377 377
q18 7733 7249 7200 7200
q19 1863 969 551 551
q20 328 339 225 225
q21 3717 3158 2346 2346
q22 1058 1013 960 960
Total cold run time: 116010 ms
Total hot run time: 33782 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5285 5053 5029 5029
q2 240 337 236 236
q3 2141 2663 2265 2265
q4 1324 1760 1387 1387
q5 4477 4454 4426 4426
q6 212 165 130 130
q7 1972 1891 1772 1772
q8 2593 2545 2537 2537
q9 7244 7346 7014 7014
q10 3058 3176 2748 2748
q11 567 507 502 502
q12 687 758 625 625
q13 3655 3902 3245 3245
q14 300 334 314 314
q15 535 475 472 472
q16 457 492 454 454
q17 1175 1519 1398 1398
q18 7935 7654 7540 7540
q19 812 829 844 829
q20 1977 2024 1859 1859
q21 4901 4506 4452 4452
q22 1092 1068 1054 1054
Total cold run time: 52639 ms
Total hot run time: 50288 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3116416142
# FE UT Coverage Report
Increment line coverage ` 84.54% (339/401)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_c3f0b61e7e8cc8de1b85cb8c4ac561353c78b15d/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_c3f0b61e7e8cc8de1b85cb8c4ac561353c78b15d/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3076589765
<details>
<summary>TPC-H: <b>Total hot run time: 33924 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79a5db32079ea3bcb6f30e554e2924fe379da810, data reload: false
------ Round 1 ----------------------------------
q1 17612 5240 5111 5111
q2 1920 286 184 184
q3 10310 1371 705 705
q4 10237 1009 519 519
q5 7865 2284 2334 2284
q6 187 159 130 130
q7 910 761 583 583
q8 9307 1302 1043 1043
q9 7106 5162 5086 5086
q10 6889 2367 1957 1957
q11 488 287 282 282
q12 340 354 213 213
q13 17753 3629 3033 3033
q14 232 225 224 224
q15 541 483 495 483
q16 421 421 373 373
q17 599 878 347 347
q18 7518 7225 7254 7225
q19 1224 950 593 593
q20 342 332 220 220
q21 3746 2543 2386 2386
q22 1012 1009 943 943
Total cold run time: 106559 ms
Total hot run time: 33924 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5196 5209 5190 5190
q2 236 318 217 217
q3 2207 2649 2291 2291
q4 1351 1767 1323 1323
q5 4206 4497 4457 4457
q6 222 172 124 124
q7 2068 2011 1810 1810
q8 2647 2575 2555 2555
q9 7392 7268 7359 7268
q10 3106 3318 2869 2869
q11 561 639 519 519
q12 656 794 600 600
q13 3603 4009 3399 3399
q14 292 303 299 299
q15 520 489 481 481
q16 447 520 449 449
q17 1242 1675 1329 1329
q18 7949 7747 7775 7747
q19 840 857 1040 857
q20 1938 2021 1906 1906
q21 4838 4416 4345 4345
q22 1105 1040 1005 1005
Total cold run time: 52622 ms
Total hot run time: 51040 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3078999551
# FE UT Coverage Report
Increment line coverage ` 85.25% (312/366)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49982_7dd3d7e2d7f78983d969a79698664d3686e15112/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49982_7dd3d7e2d7f78983d969a79698664d3686e15112/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3078706183
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3078793695
<details>
<summary>TPC-H: <b>Total hot run time: 34296 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7dd3d7e2d7f78983d969a79698664d3686e15112, data reload: false
------ Round 1 ----------------------------------
q1 17556 5260 5349 5260
q2 1930 284 181 181
q3 10307 1267 730 730
q4 10228 1007 543 543
q5 7469 2408 2373 2373
q6 182 164 132 132
q7 931 753 618 618
q8 9327 1255 1086 1086
q9 6828 5088 5111 5088
q10 6886 2387 1960 1960
q11 476 284 273 273
q12 341 352 213 213
q13 17744 3696 3084 3084
q14 225 227 219 219
q15 558 479 496 479
q16 423 448 378 378
q17 593 881 347 347
q18 7592 7362 7228 7228
q19 1097 949 548 548
q20 354 342 219 219
q21 4078 3170 2362 2362
q22 1045 1016 975 975
Total cold run time: 106170 ms
Total hot run time: 34296 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5216 5167 5139 5139
q2 242 317 222 222
q3 2187 2653 2332 2332
q4 1336 1767 1314 1314
q5 4221 4541 4596 4541
q6 212 164 127 127
q7 1989 1962 1803 1803
q8 2654 2566 2580 2566
q9 7414 7251 7313 7251
q10 3130 3308 2915 2915
q11 577 510 496 496
q12 692 780 615 615
q13 3592 4052 3400 3400
q14 296 307 295 295
q15 509 477 532 477
q16 477 495 455 455
q17 1230 1543 1396 1396
q18 7888 7703 7574 7574
q19 839 844 951 844
q20 2015 2064 2036 2036
q21 4961 4641 4486 4486
q22 1286 1060 1022 1022
Total cold run time: 52963 ms
Total hot run time: 51306 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on code in PR #49982:
URL: https://github.com/apache/doris/pull/49982#discussion_r2209067119
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/TreeNode.java:
##########
@@ -150,22 +150,20 @@ default NODE_TYPE rewriteDownShortCircuit(Function<NODE_TYPE, NODE_TYPE> rewrite
/**
* similar to rewriteDownShortCircuit, except that only subtrees, whose root satisfies
- * border predicate are rewritten.
+ * border predicate are rewritten, and if a node not match predicate, then its descendant will not rewrite.
*/
default NODE_TYPE rewriteDownShortCircuitDown(Function<NODE_TYPE, NODE_TYPE> rewriteFunction,
- Predicate border, boolean aboveBorder) {
+ Predicate predicate) {
Review Comment:
this function no usage before, and the second arg is not friendly for use, so this pr reuse this function and delete the second arg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](nereids) Add constant propagation [doris]
Posted by "yujun777 (via GitHub)" <gi...@apache.org>.
yujun777 commented on PR #49982:
URL: https://github.com/apache/doris/pull/49982#issuecomment-3100689447
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org