You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "seawinde (via GitHub)" <gi...@apache.org> on 2024/02/05 13:02:28 UTC
[PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
seawinde opened a new pull request, #30870:
URL: https://github.com/apache/doris/pull/30870
## Proposed changes
query rewrite by materialized view sql is as following
q1, q5, q6, q8, q9, q12, q14
the other is not supported now, will be supported later
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927217005
<details>
<summary>TPC-H: <b>Total hot run time: 37058 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cd62c6321b465df1b87c100455855c46a81e8543, data reload: false
------ Round 1 ----------------------------------
q1 17637 4794 4360 4360
q2 2043 143 134 134
q3 10589 922 976 922
q4 4641 784 745 745
q5 7670 2894 2825 2825
q6 183 121 119 119
q7 1165 742 736 736
q8 9271 2047 2068 2047
q9 7254 6410 6403 6403
q10 8078 2397 2432 2397
q11 419 210 213 210
q12 778 279 270 270
q13 18016 3329 3284 3284
q14 270 243 254 243
q15 529 498 484 484
q16 470 395 410 395
q17 956 534 507 507
q18 6820 5940 5971 5940
q19 1588 1385 1425 1385
q20 627 339 320 320
q21 7125 3138 3042 3042
q22 801 309 290 290
Total cold run time: 106930 ms
Total hot run time: 37058 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4452 4434 4465 4434
q2 334 228 242 228
q3 3012 2839 2853 2839
q4 1890 1688 1645 1645
q5 5248 5253 5192 5192
q6 191 115 115 115
q7 2100 1767 1817 1767
q8 3128 3215 3267 3215
q9 8308 8314 8336 8314
q10 5763 3572 3572 3572
q11 530 482 460 460
q12 724 571 586 571
q13 12906 3065 3088 3065
q14 273 254 279 254
q15 536 492 494 492
q16 509 472 474 472
q17 1912 1735 1727 1727
q18 8114 7579 7482 7482
q19 9722 1538 1590 1538
q20 2122 1912 1914 1912
q21 4723 4621 4716 4621
q22 556 470 456 456
Total cold run time: 77053 ms
Total hot run time: 54371 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #30870:
URL: https://github.com/apache/doris/pull/30870
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952339946
<details>
<summary>TPC-H: <b>Total hot run time: 41836 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2235d8e887bcc2b31f14b7f5ec4031760fb065c, data reload: false
------ Round 1 ----------------------------------
q1 18539 5205 5126 5126
q2 2519 149 145 145
q3 11517 1050 1008 1008
q4 4655 983 1000 983
q5 7698 3220 3273 3220
q6 196 137 137 137
q7 1274 782 757 757
q8 9241 2094 2082 2082
q9 7627 6699 6678 6678
q10 8314 2664 2646 2646
q11 422 211 207 207
q12 763 335 336 335
q13 18001 3667 3694 3667
q14 299 261 252 252
q15 620 500 500 500
q16 483 409 410 409
q17 942 829 828 828
q18 7299 6734 6683 6683
q19 1546 1495 1502 1495
q20 539 373 338 338
q21 6450 4026 3993 3993
q22 425 356 347 347
Total cold run time: 109369 ms
Total hot run time: 41836 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4849 4825 4825 4825
q2 297 185 185 185
q3 3637 3599 3582 3582
q4 2598 2541 2543 2541
q5 5794 5766 5781 5766
q6 209 128 130 128
q7 2264 1641 1650 1641
q8 3043 3128 3152 3128
q9 8722 8775 8733 8733
q10 4158 4236 4246 4236
q11 512 386 369 369
q12 771 550 555 550
q13 4184 3375 3396 3375
q14 274 235 238 235
q15 638 515 490 490
q16 482 443 444 443
q17 1674 1619 1612 1612
q18 8376 7661 7646 7646
q19 1630 1634 1638 1634
q20 2087 1812 1821 1812
q21 6771 6335 6334 6334
q22 578 507 517 507
Total cold run time: 63548 ms
Total hot run time: 59772 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1481165869
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVCache.java:
##########
@@ -68,13 +77,28 @@ public static MTMVCache from(MTMV mtmv, ConnectContext connectContext) {
}
Plan mvRewrittenPlan =
planner.plan(unboundMvPlan, PhysicalProperties.ANY, ExplainLevel.REWRITTEN_PLAN);
- Plan mvPlan = mvRewrittenPlan instanceof LogicalResultSink
- ? (Plan) ((LogicalResultSink) mvRewrittenPlan).child() : mvRewrittenPlan;
+ // remove top sort and result sink node above materialized view plan
+ if (mvRewrittenPlan instanceof LogicalResultSink) {
+ // replace result sink to table sink for sort eliminate
+ mvRewrittenPlan = new LogicalOlapTableSink<>(new Database(), mtmv, mtmv.getBaseSchema(),
+ mtmv.getPartitionIds(),
+ mvRewrittenPlan.getOutput().stream().map(NamedExpression.class::cast).collect(Collectors.toList()),
+ mtmv.isPartitionedTable(), DMLCommandType.NONE, mvRewrittenPlan.child(0));
+ // eliminate sort under sink because sort is useless for materialized view
+ planner.getCascadesContext().setRewritePlan(mvRewrittenPlan);
+ Rewriter rewriter = Rewriter.getCteChildrenRewriter(planner.getCascadesContext(),
+ Lists.newArrayList(AbstractBatchJobExecutor.topDown(new EliminateSortUnderSink())));
+ rewriter.execute();
+ }
+ if (planner.getCascadesContext().getRewritePlan() instanceof Sink) {
+ // remove the sink
+ mvRewrittenPlan = planner.getCascadesContext().getRewritePlan().child(0);
+ }
Review Comment:
should notice, result sink is not always the top plan node after plan. so u should replace result sink with table sink before plan. and then when u do plan, the top sort node will be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952397688
<details>
<summary>TPC-DS: <b>Total hot run time: 185903 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 019210532f487f5c0d52755d91fc5040873c0112, data reload: false
query1 922 353 354 353
query2 6521 1740 1718 1718
query3 6693 208 206 206
query4 23116 21197 21119 21119
query5 4284 463 398 398
query6 246 168 163 163
query7 4614 290 293 290
query8 248 194 207 194
query9 8448 2869 2832 2832
query10 419 231 223 223
query11 15123 14565 14694 14565
query12 136 82 82 82
query13 1704 447 435 435
query14 9080 7644 7768 7644
query15 211 191 193 191
query16 7429 264 249 249
query17 1408 566 530 530
query18 1964 269 266 266
query19 194 150 151 150
query20 87 83 86 83
query21 191 124 113 113
query22 4935 4735 4800 4735
query23 32511 31515 31510 31510
query24 12867 3445 3410 3410
query25 654 371 366 366
query26 1894 153 165 153
query27 3048 320 322 320
query28 6696 1832 1819 1819
query29 1168 621 617 617
query30 284 136 143 136
query31 944 748 751 748
query32 96 63 54 54
query33 717 244 238 238
query34 1076 495 501 495
query35 947 824 828 824
query36 941 881 882 881
query37 266 61 64 61
query38 3298 3190 3207 3190
query39 1367 1348 1318 1318
query40 284 108 105 105
query41 38 36 39 36
query42 110 101 104 101
query43 471 435 442 435
query44 1066 686 701 686
query45 205 181 180 180
query46 1037 790 785 785
query47 1635 1584 1595 1584
query48 434 345 352 345
query49 1221 307 301 301
query50 776 381 395 381
query51 5279 5148 5180 5148
query52 102 90 95 90
query53 389 299 302 299
query54 296 217 235 217
query55 87 82 83 82
query56 222 200 201 200
query57 1041 944 974 944
query58 225 199 199 199
query59 2254 2078 2143 2078
query60 261 224 238 224
query61 84 85 80 80
query62 589 370 371 370
query63 345 304 294 294
query64 6392 3047 3121 3047
query65 3306 3270 3254 3254
query66 1352 326 325 325
query67 14540 14246 14248 14246
query68 4068 539 546 539
query69 507 354 368 354
query70 1188 1210 1237 1210
query71 301 259 257 257
query72 5997 2782 2610 2610
query73 683 308 314 308
query74 6903 6416 6468 6416
query75 3164 2568 2548 2548
query76 2753 1106 1193 1106
query77 352 244 240 240
query78 9421 8851 8780 8780
query79 966 510 507 507
query80 528 356 347 347
query81 437 208 202 202
query82 235 83 85 83
query83 146 122 120 120
query84 235 77 81 77
query85 1047 351 351 351
query86 296 307 309 307
query87 3477 3301 3303 3301
query88 2710 2268 2277 2268
query89 448 361 369 361
query90 1897 169 169 169
query91 160 130 126 126
query92 57 51 47 47
query93 993 508 504 504
query94 1181 191 174 174
query95 480 8721 8448 8448
query96 586 273 269 269
query97 4459 4270 4252 4252
query98 218 198 202 198
query99 1088 709 720 709
Total cold run time: 267769 ms
Total hot run time: 185903 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952406747
<details>
<summary>ClickBench: <b>Total hot run time: 31.04 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 019210532f487f5c0d52755d91fc5040873c0112, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.22 0.08 0.08
query4 1.67 0.09 0.08
query5 0.49 0.48 0.47
query6 1.35 0.61 0.61
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.53 0.45 0.45
query10 0.47 0.48 0.48
query11 0.12 0.09 0.10
query12 0.12 0.10 0.10
query13 0.59 0.59 0.59
query14 0.76 0.79 0.78
query15 0.82 0.81 0.79
query16 0.33 0.33 0.33
query17 0.90 0.88 0.90
query18 0.16 0.15 0.19
query19 1.74 1.68 1.71
query20 0.02 0.01 0.01
query21 15.41 0.62 0.60
query22 2.71 3.89 2.42
query23 17.55 0.93 1.09
query24 2.08 0.49 0.37
query25 0.62 0.07 0.06
query26 0.16 0.14 0.14
query27 0.07 0.05 0.05
query28 11.97 0.86 0.82
query29 12.66 3.35 3.35
query30 0.51 0.50 0.51
query31 2.96 0.37 0.37
query32 3.62 0.47 0.48
query33 3.14 3.12 3.13
query34 15.37 4.51 4.52
query35 4.48 4.50 4.47
query36 1.06 0.95 0.94
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.01 0.01 0.02
query40 0.16 0.15 0.14
query41 0.07 0.01 0.01
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.21 s
Total hot run time: 31.04 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1926950648
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1929166608
<details>
<summary>TPC-DS: <b>Total hot run time: 174866 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 71145d5da6824d1040030eceaf75abc3a12aff2c, data reload: false
query1 927 349 331 331
query2 6540 2029 1989 1989
query3 6717 210 210 210
query4 31693 21944 22129 21944
query5 4330 424 352 352
query6 246 169 158 158
query7 4616 275 271 271
query8 234 182 181 181
query9 9036 2283 2277 2277
query10 418 208 210 208
query11 17708 15382 15380 15380
query12 129 75 75 75
query13 1616 421 413 413
query14 9524 6910 7335 6910
query15 246 172 174 172
query16 8149 271 246 246
query17 2057 538 503 503
query18 2103 291 272 272
query19 342 145 139 139
query20 82 78 84 78
query21 199 120 122 120
query22 4912 4662 4638 4638
query23 30906 30165 30050 30050
query24 10480 2762 2741 2741
query25 539 349 330 330
query26 713 147 148 147
query27 2216 301 297 297
query28 5884 1844 1848 1844
query29 850 613 625 613
query30 286 134 137 134
query31 913 706 735 706
query32 95 50 54 50
query33 732 221 219 219
query34 1127 462 465 462
query35 879 799 770 770
query36 1038 928 908 908
query37 95 54 60 54
query38 3209 3132 3101 3101
query39 1307 1262 1236 1236
query40 184 96 92 92
query41 41 39 34 34
query42 98 88 94 88
query43 540 492 473 473
query44 1070 693 697 693
query45 187 182 175 175
query46 1042 671 661 661
query47 1593 1421 1543 1421
query48 432 360 351 351
query49 1035 289 291 289
query50 772 376 382 376
query51 5297 5159 5204 5159
query52 95 88 83 83
query53 332 272 274 272
query54 287 206 212 206
query55 79 74 76 74
query56 220 196 199 196
query57 993 943 885 885
query58 201 183 183 183
query59 2608 2425 2477 2425
query60 237 207 210 207
query61 84 84 91 84
query62 666 353 347 347
query63 295 263 267 263
query64 4901 3621 3610 3610
query65 3264 3266 3234 3234
query66 826 308 310 308
query67 14357 14194 14140 14140
query68 4526 527 524 524
query69 467 315 320 315
query70 1292 1225 1270 1225
query71 317 245 248 245
query72 6075 2859 2743 2743
query73 701 333 328 328
query74 6539 6157 6351 6157
query75 3035 2390 2385 2385
query76 3293 935 913 913
query77 353 233 223 223
query78 9200 8633 8530 8530
query79 2957 488 500 488
query80 2088 351 340 340
query81 549 198 197 197
query82 859 80 81 80
query83 245 130 122 122
query84 281 86 78 78
query85 2206 341 338 338
query86 472 284 305 284
query87 3385 3295 3243 3243
query88 4228 2315 2319 2315
query89 462 373 371 371
query90 1975 164 164 164
query91 154 122 123 122
query92 57 41 44 41
query93 4827 498 450 450
query94 1288 178 176 176
query95 469 7887 354 354
query96 608 286 283 283
query97 4220 4104 4099 4099
query98 220 208 191 191
query99 1139 684 716 684
Total cold run time: 283316 ms
Total hot run time: 174866 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952377437
<details>
<summary>TPC-H: <b>Total hot run time: 41283 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 019210532f487f5c0d52755d91fc5040873c0112, data reload: false
------ Round 1 ----------------------------------
q1 17580 4909 4838 4838
q2 2038 142 141 141
q3 10745 1046 1011 1011
q4 4653 948 957 948
q5 7714 3241 3271 3241
q6 194 131 130 130
q7 1256 779 777 777
q8 9254 2087 2071 2071
q9 7645 6688 6669 6669
q10 8311 2660 2624 2624
q11 419 229 222 222
q12 718 333 331 331
q13 17993 3677 3653 3653
q14 295 265 268 265
q15 585 504 536 504
q16 470 407 418 407
q17 924 869 840 840
q18 7345 6681 6572 6572
q19 1539 1512 1491 1491
q20 639 364 329 329
q21 6680 3875 3933 3875
q22 859 355 344 344
Total cold run time: 107856 ms
Total hot run time: 41283 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4816 4798 4808 4798
q2 297 182 189 182
q3 3592 3584 3576 3576
q4 2531 2510 2499 2499
q5 5776 5731 5765 5731
q6 211 123 129 123
q7 2247 1641 1668 1641
q8 3022 3081 3093 3081
q9 8684 8708 8648 8648
q10 6915 4235 4241 4235
q11 525 384 356 356
q12 764 534 534 534
q13 4173 3409 3435 3409
q14 265 232 232 232
q15 590 507 502 502
q16 480 432 420 420
q17 1684 1615 1597 1597
q18 8298 7565 7597 7565
q19 1624 1631 1626 1626
q20 2115 1852 1811 1811
q21 6507 6129 6105 6105
q22 571 513 524 513
Total cold run time: 65687 ms
Total hot run time: 59184 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952412178
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 019210532f487f5c0d52755d91fc5040873c0112 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.3 seconds inserted 10000000 Rows, about 751K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927253947
<details>
<summary>ClickBench: <b>Total hot run time: 30.7 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cd62c6321b465df1b87c100455855c46a81e8543, data reload: false
query1 0.04 0.03 0.04
query2 0.05 0.02 0.02
query3 0.23 0.06 0.06
query4 1.67 0.10 0.10
query5 0.56 0.51 0.51
query6 1.19 0.67 0.63
query7 0.02 0.01 0.01
query8 0.04 0.02 0.02
query9 0.55 0.50 0.51
query10 0.56 0.56 0.54
query11 0.12 0.09 0.09
query12 0.11 0.08 0.09
query13 0.61 0.62 0.61
query14 0.79 0.80 0.81
query15 0.80 0.78 0.77
query16 0.39 0.39 0.40
query17 1.00 0.96 0.97
query18 0.22 0.25 0.26
query19 1.93 1.78 1.74
query20 0.01 0.01 0.01
query21 15.40 0.57 0.56
query22 2.63 2.57 1.76
query23 17.19 0.91 0.82
query24 2.42 1.36 0.87
query25 0.34 0.22 0.11
query26 0.29 0.14 0.13
query27 0.04 0.05 0.05
query28 11.60 0.85 0.83
query29 12.54 3.08 3.18
query30 0.65 0.53 0.54
query31 2.79 0.34 0.35
query32 3.37 0.48 0.48
query33 3.29 3.24 3.17
query34 15.90 4.30 4.29
query35 4.34 4.27 4.27
query36 1.11 1.04 1.07
query37 0.07 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.01 0.02
query40 0.17 0.13 0.12
query41 0.06 0.01 0.02
query42 0.03 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 105.19 s
Total hot run time: 30.7 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952345585
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1495093937
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVCache.java:
##########
@@ -68,13 +77,28 @@ public static MTMVCache from(MTMV mtmv, ConnectContext connectContext) {
}
Plan mvRewrittenPlan =
planner.plan(unboundMvPlan, PhysicalProperties.ANY, ExplainLevel.REWRITTEN_PLAN);
- Plan mvPlan = mvRewrittenPlan instanceof LogicalResultSink
- ? (Plan) ((LogicalResultSink) mvRewrittenPlan).child() : mvRewrittenPlan;
+ // remove top sort and result sink node above materialized view plan
+ if (mvRewrittenPlan instanceof LogicalResultSink) {
+ // replace result sink to table sink for sort eliminate
+ mvRewrittenPlan = new LogicalOlapTableSink<>(new Database(), mtmv, mtmv.getBaseSchema(),
+ mtmv.getPartitionIds(),
+ mvRewrittenPlan.getOutput().stream().map(NamedExpression.class::cast).collect(Collectors.toList()),
+ mtmv.isPartitionedTable(), DMLCommandType.NONE, mvRewrittenPlan.child(0));
+ // eliminate sort under sink because sort is useless for materialized view
+ planner.getCascadesContext().setRewritePlan(mvRewrittenPlan);
+ Rewriter rewriter = Rewriter.getCteChildrenRewriter(planner.getCascadesContext(),
+ Lists.newArrayList(AbstractBatchJobExecutor.topDown(new EliminateSortUnderSink())));
+ rewriter.execute();
+ }
+ if (planner.getCascadesContext().getRewritePlan() instanceof Sink) {
+ // remove the sink
+ mvRewrittenPlan = planner.getCascadesContext().getRewritePlan().child(0);
+ }
Review Comment:
Have fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1529608877
##########
fe/fe-common/src/main/java/org/apache/doris/common/Value.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/fe/src/main/java/org/apache/impala/Pair.java
+// and modified by Doris
Review Comment:
have removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-2002301896
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927242279
<details>
<summary>TPC-DS: <b>Total hot run time: 181675 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cd62c6321b465df1b87c100455855c46a81e8543, data reload: false
query1 923 351 339 339
query2 6545 2052 1893 1893
query3 6693 215 208 208
query4 31881 21957 21948 21948
query5 4277 371 423 371
query6 255 166 161 161
query7 4606 291 282 282
query8 238 169 185 169
query9 8991 2280 2267 2267
query10 416 209 216 209
query11 18284 15426 15395 15395
query12 123 74 75 74
query13 1622 409 419 409
query14 9421 7086 6923 6923
query15 281 169 179 169
query16 8149 274 248 248
query17 2030 539 488 488
query18 2101 269 266 266
query19 357 148 137 137
query20 81 74 75 74
query21 192 129 121 121
query22 4937 4627 4681 4627
query23 31343 30124 30057 30057
query24 10090 2763 2768 2763
query25 540 339 340 339
query26 710 143 150 143
query27 2178 304 300 300
query28 5708 1847 1825 1825
query29 868 609 607 607
query30 281 133 138 133
query31 915 696 699 696
query32 87 55 49 49
query33 635 222 215 215
query34 841 458 461 458
query35 858 758 775 758
query36 1055 942 881 881
query37 101 57 63 57
query38 3277 3108 3160 3108
query39 1309 1268 1256 1256
query40 189 99 95 95
query41 36 39 34 34
query42 100 89 88 88
query43 532 463 497 463
query44 1039 699 700 699
query45 192 180 175 175
query46 1042 656 673 656
query47 1626 1549 1480 1480
query48 426 351 344 344
query49 1047 285 299 285
query50 755 370 367 367
query51 5256 5076 5229 5076
query52 102 88 88 88
query53 331 268 279 268
query54 289 214 217 214
query55 78 80 78 78
query56 222 201 191 191
query57 988 916 887 887
query58 191 181 180 180
query59 2355 2301 2263 2263
query60 244 223 210 210
query61 82 80 83 80
query62 621 370 361 361
query63 284 266 286 266
query64 4713 3590 3580 3580
query65 3241 3248 3252 3248
query66 827 325 303 303
query67 14627 14109 13970 13970
query68 4432 534 537 534
query69 466 321 330 321
query70 1246 1177 1257 1177
query71 307 241 270 241
query72 6061 2825 2670 2670
query73 717 331 321 321
query74 6669 6253 6176 6176
query75 3000 2360 2367 2360
query76 2997 901 1004 901
query77 357 228 227 227
query78 9178 8599 8543 8543
query79 3079 491 498 491
query80 2075 359 330 330
query81 551 198 195 195
query82 826 90 85 85
query83 246 130 124 124
query84 283 84 80 80
query85 2149 352 324 324
query86 493 283 303 283
query87 3363 3194 3197 3194
query88 3978 2327 2328 2327
query89 460 373 363 363
query90 1905 160 162 160
query91 151 117 124 117
query92 51 44 43 43
query93 4289 495 486 486
query94 1288 175 177 175
query95 8133 7816 7921 7816
query96 595 267 279 267
query97 4286 4095 4166 4095
query98 221 199 190 190
query99 1125 741 687 687
Total cold run time: 289611 ms
Total hot run time: 181675 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1929176252
<details>
<summary>ClickBench: <b>Total hot run time: 31.24 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 71145d5da6824d1040030eceaf75abc3a12aff2c, data reload: false
query1 0.04 0.04 0.03
query2 0.06 0.03 0.03
query3 0.23 0.07 0.06
query4 1.64 0.10 0.10
query5 0.53 0.52 0.52
query6 1.21 0.64 0.62
query7 0.02 0.01 0.02
query8 0.04 0.02 0.02
query9 0.55 0.49 0.51
query10 0.56 0.54 0.55
query11 0.12 0.08 0.08
query12 0.11 0.08 0.09
query13 0.62 0.60 0.59
query14 0.78 0.78 0.81
query15 0.81 0.78 0.78
query16 0.39 0.39 0.36
query17 1.02 0.98 1.02
query18 0.21 0.24 0.25
query19 1.84 1.77 1.80
query20 0.01 0.01 0.01
query21 15.45 0.57 0.58
query22 2.78 2.49 2.30
query23 17.17 0.77 0.73
query24 2.42 1.09 0.91
query25 0.26 0.28 0.05
query26 0.58 0.13 0.13
query27 0.06 0.04 0.06
query28 11.60 0.86 0.83
query29 12.51 3.17 3.15
query30 0.65 0.53 0.53
query31 2.78 0.35 0.36
query32 3.32 0.48 0.47
query33 3.28 3.18 3.27
query34 15.81 4.35 4.28
query35 4.30 4.31 4.32
query36 1.11 1.06 1.06
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.03 0.02 0.01
query40 0.17 0.13 0.16
query41 0.07 0.02 0.01
query42 0.03 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.31 s
Total hot run time: 31.24 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1929145890
<details>
<summary>TPC-H: <b>Total hot run time: 37115 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 71145d5da6824d1040030eceaf75abc3a12aff2c, data reload: false
------ Round 1 ----------------------------------
q1 17945 4539 4512 4512
q2 2564 157 144 144
q3 11199 964 952 952
q4 4948 729 696 696
q5 7709 2899 2781 2781
q6 181 123 124 123
q7 1188 723 709 709
q8 9312 2026 2017 2017
q9 7315 6353 6387 6353
q10 8060 2422 2415 2415
q11 419 213 210 210
q12 735 260 280 260
q13 18005 3329 3305 3305
q14 277 264 255 255
q15 528 507 489 489
q16 466 406 405 405
q17 926 564 484 484
q18 6892 6081 6003 6003
q19 1587 1374 1408 1374
q20 610 370 328 328
q21 7160 3094 3004 3004
q22 806 302 296 296
Total cold run time: 108832 ms
Total hot run time: 37115 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4541 4366 4406 4366
q2 334 243 221 221
q3 2992 2892 2817 2817
q4 1897 1764 1609 1609
q5 5163 5256 5253 5253
q6 188 115 116 115
q7 2147 1747 1752 1747
q8 3172 3233 3522 3233
q9 8328 8283 8255 8255
q10 5792 3556 3571 3556
q11 560 469 458 458
q12 769 578 600 578
q13 16896 3060 3114 3060
q14 284 263 272 263
q15 532 490 490 490
q16 519 477 455 455
q17 1858 1693 1648 1648
q18 7849 7772 7537 7537
q19 1578 1515 1479 1479
q20 2102 1919 1915 1915
q21 4713 4524 4651 4524
q22 527 478 461 461
Total cold run time: 72741 ms
Total hot run time: 54040 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927110676
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1953593909
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-2005705352
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1527777531
##########
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVCache.java:
##########
@@ -62,13 +75,27 @@ public static MTMVCache from(MTMV mtmv, ConnectContext connectContext) {
mvSqlStatementContext.getConnectContext().setStatementContext(mvSqlStatementContext);
}
Plan originPlan = planner.plan(unboundMvPlan, PhysicalProperties.ANY, ExplainLevel.REWRITTEN_PLAN);
+ // change result sink to table sink for eliminate sort
Plan mvPlan = originPlan.accept(new DefaultPlanRewriter<Object>() {
@Override
- public Plan visit(Plan plan, Object context) {
- if (plan instanceof LogicalResultSink) {
- return ((LogicalResultSink<Plan>) plan).child();
- }
- return super.visit(plan, context);
+ public Plan visitLogicalResultSink(LogicalResultSink<? extends Plan> logicalResultSink, Object context) {
+ return new LogicalOlapTableSink<>(new Database(), mtmv, mtmv.getBaseSchema(),
+ mtmv.getPartitionIds(),
+ logicalResultSink.getOutput()
+ .stream().map(NamedExpression.class::cast).collect(Collectors.toList()),
+ mtmv.isPartitionedTable(), DMLCommandType.NONE, logicalResultSink.child());
Review Comment:
why not replace result sink to unbound table sink on `unboundMvPlan` to avoid call ELIMINATE_SORT explicity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1996830794
<details>
<summary>TPC-H: <b>Total hot run time: 37897 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa45998b1429a78180ce8d36886b4747f97d11a3, data reload: false
------ Round 1 ----------------------------------
q1 17622 4195 4087 4087
q2 2015 148 140 140
q3 10615 1062 872 872
q4 6380 714 718 714
q5 7484 2555 2655 2555
q6 181 123 124 123
q7 1229 808 792 792
q8 9361 1950 2036 1950
q9 7087 6394 6433 6394
q10 8459 3494 3598 3494
q11 429 227 219 219
q12 703 301 311 301
q13 17784 2835 2854 2835
q14 267 250 245 245
q15 506 452 444 444
q16 512 392 385 385
q17 943 563 561 561
q18 7121 6382 6338 6338
q19 1694 1482 1378 1378
q20 538 294 267 267
q21 6599 3549 3508 3508
q22 350 295 309 295
Total cold run time: 107879 ms
Total hot run time: 37897 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4064 4038 4098 4038
q2 322 219 220 219
q3 2907 2838 2739 2739
q4 1827 1520 1532 1520
q5 5193 5189 5218 5189
q6 196 114 118 114
q7 2224 1810 1871 1810
q8 3137 3257 3285 3257
q9 8505 8519 8490 8490
q10 3671 3652 3642 3642
q11 541 443 446 443
q12 722 548 535 535
q13 16926 2844 2828 2828
q14 272 241 256 241
q15 483 446 441 441
q16 456 429 416 416
q17 1720 1486 1471 1471
q18 7493 7218 6997 6997
q19 1620 1554 1519 1519
q20 1887 1691 1696 1691
q21 4848 4801 4647 4647
q22 531 447 479 447
Total cold run time: 69545 ms
Total hot run time: 52694 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-2008614001
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-2002308329
<details>
<summary>TPC-H: <b>Total hot run time: 39262 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 160fd37acc5d2f62591afac9c2733e0557721f75, data reload: false
------ Round 1 ----------------------------------
q1 17635 4353 4246 4246
q2 2016 162 151 151
q3 10579 1082 912 912
q4 7429 726 723 723
q5 7469 2818 2776 2776
q6 198 125 124 124
q7 1233 868 849 849
q8 9346 2168 2153 2153
q9 7348 6644 6630 6630
q10 8511 3590 3697 3590
q11 426 237 219 219
q12 799 323 314 314
q13 17996 2887 2856 2856
q14 287 267 251 251
q15 516 467 461 461
q16 507 397 394 394
q17 996 531 631 531
q18 7200 6499 6413 6413
q19 3474 1581 1502 1502
q20 548 295 285 285
q21 6223 3587 3602 3587
q22 357 295 321 295
Total cold run time: 111093 ms
Total hot run time: 39262 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4246 4280 4338 4280
q2 335 225 224 224
q3 3009 2877 2867 2867
q4 1893 1540 1593 1540
q5 5302 5293 5321 5293
q6 209 115 117 115
q7 2279 1849 1836 1836
q8 3304 3483 3463 3463
q9 8674 8596 8674 8596
q10 3775 3740 3700 3700
q11 565 444 446 444
q12 757 538 561 538
q13 16876 2871 2851 2851
q14 279 242 259 242
q15 494 451 444 444
q16 473 418 414 414
q17 1863 1574 1523 1523
q18 7503 7190 7167 7167
q19 1780 1680 1638 1638
q20 1955 1706 1711 1706
q21 4951 4709 4831 4709
q22 542 447 459 447
Total cold run time: 71064 ms
Total hot run time: 54037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1929102252
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1503562426
##########
fe/fe-common/src/main/java/org/apache/doris/common/Value.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/fe/src/main/java/org/apache/impala/Pair.java
+// and modified by Doris
Review Comment:
i think this is not right? need remove or update it?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSink.java:
##########
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Eliminate sort under sink, this can be used when sort is insensitive to the result
+ */
+public class EliminateSortUnderSink implements RewriteRuleFactory {
Review Comment:
add this rule to rewriter directly, not only used in Mtmv
##########
fe/fe-common/src/main/java/org/apache/doris/common/Value.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/fe/src/main/java/org/apache/impala/Pair.java
+// and modified by Doris
+
+package org.apache.doris.common;
+
+import java.util.Objects;
+
+/**
+ * The object which contains a value, it also can be used to wrap a private variable
+ */
+public class Value<O> {
Review Comment:
why need Value class? what benifit from it?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSink.java:
##########
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Eliminate sort under sink, this can be used when sort is insensitive to the result
+ */
+public class EliminateSortUnderSink implements RewriteRuleFactory {
+
+ @Override
+ public List<Rule> buildRules() {
+ return ImmutableList.of(
+ RuleType.ELIMINATE_SORT_UNDER_OLAP_TABLE_SINK.build(
Review Comment:
add a new class named TableSink, and let OlapTableSink extends it. so all other table sink could apply this rule
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSink.java:
##########
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Eliminate sort under sink, this can be used when sort is insensitive to the result
+ */
+public class EliminateSortUnderSink implements RewriteRuleFactory {
Review Comment:
i think u should update `EliminateSort`, not add a new rule.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1952308520
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1527409975
##########
fe/fe-common/src/main/java/org/apache/doris/common/Value.java:
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// This file is copied from
+// https://github.com/apache/impala/blob/branch-2.9.0/fe/src/main/java/org/apache/impala/Pair.java
+// and modified by Doris
+
+package org.apache.doris.common;
+
+import java.util.Objects;
+
+/**
+ * The object which contains a value, it also can be used to wrap a private variable
+ */
+public class Value<O> {
Review Comment:
have removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1929181740
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 71145d5da6824d1040030eceaf75abc3a12aff2c with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 13.2 seconds inserted 10000000 Rows, about 757K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1926995930
<details>
<summary>TPC-H: <b>Total hot run time: 36551 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35bf3e371e510340d7342d55f1614aaba6dd1dd5, data reload: false
------ Round 1 ----------------------------------
q1 17647 5052 4420 4420
q2 2033 137 134 134
q3 10681 944 886 886
q4 4661 706 717 706
q5 7678 2768 2751 2751
q6 181 120 121 120
q7 1160 720 699 699
q8 9251 2178 1991 1991
q9 7183 6342 6291 6291
q10 8100 2436 2406 2406
q11 420 194 197 194
q12 718 269 276 269
q13 17998 3266 3260 3260
q14 283 248 250 248
q15 524 492 483 483
q16 481 399 398 398
q17 918 531 509 509
q18 6783 5992 5763 5763
q19 1556 1454 1378 1378
q20 614 328 328 328
q21 6622 3030 3110 3030
q22 812 287 299 287
Total cold run time: 106304 ms
Total hot run time: 36551 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4437 4312 4331 4312
q2 323 228 222 222
q3 2970 2829 2800 2800
q4 1799 1672 1674 1672
q5 5173 5204 5158 5158
q6 187 113 113 113
q7 2126 1759 1740 1740
q8 3066 3198 3188 3188
q9 8324 8344 8250 8250
q10 6016 3551 3552 3551
q11 545 450 449 449
q12 751 589 568 568
q13 16337 3085 3068 3068
q14 274 258 273 258
q15 538 490 488 488
q16 514 443 480 443
q17 1866 1635 1696 1635
q18 7974 7647 7599 7599
q19 9273 1541 1530 1530
q20 2148 1906 1892 1892
q21 4778 4575 4455 4455
q22 569 463 430 430
Total cold run time: 79988 ms
Total hot run time: 53821 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927038928
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 35bf3e371e510340d7342d55f1614aaba6dd1dd5 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.5 seconds inserted 10000000 Rows, about 689K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1926949912
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927032726
<details>
<summary>ClickBench: <b>Total hot run time: 31.4 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35bf3e371e510340d7342d55f1614aaba6dd1dd5, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.02
query3 0.24 0.06 0.06
query4 1.65 0.10 0.09
query5 0.52 0.53 0.52
query6 1.27 0.65 0.65
query7 0.02 0.01 0.02
query8 0.03 0.02 0.03
query9 0.56 0.50 0.50
query10 0.54 0.56 0.55
query11 0.12 0.09 0.09
query12 0.11 0.09 0.11
query13 0.60 0.61 0.60
query14 0.78 0.80 0.80
query15 0.80 0.78 0.77
query16 0.38 0.39 0.39
query17 1.05 1.01 0.99
query18 0.25 0.25 0.25
query19 1.80 1.73 1.81
query20 0.02 0.01 0.01
query21 15.40 0.57 0.58
query22 2.52 2.97 2.11
query23 17.44 0.85 0.70
query24 2.43 1.10 1.12
query25 0.28 0.15 0.20
query26 0.61 0.14 0.13
query27 0.05 0.03 0.05
query28 11.71 0.90 0.87
query29 12.59 3.16 3.26
query30 0.59 0.56 0.54
query31 2.79 0.33 0.34
query32 3.36 0.47 0.48
query33 3.22 3.22 3.25
query34 15.81 4.29 4.26
query35 4.35 4.30 4.27
query36 1.10 1.05 1.06
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.14
query41 0.07 0.02 0.01
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.5 s
Total hot run time: 31.4 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927260576
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit cd62c6321b465df1b87c100455855c46a81e8543 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.6 seconds inserted 10000000 Rows, about 684K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1929111125
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1927022237
<details>
<summary>TPC-DS: <b>Total hot run time: 173754 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35bf3e371e510340d7342d55f1614aaba6dd1dd5, data reload: false
query1 940 345 328 328
query2 6552 1884 1868 1868
query3 6697 208 202 202
query4 31822 21953 21964 21953
query5 4266 352 351 351
query6 249 184 160 160
query7 4606 277 274 274
query8 245 171 179 171
query9 9066 2246 2222 2222
query10 419 226 215 215
query11 18524 15302 15204 15204
query12 135 74 73 73
query13 1623 410 418 410
query14 9429 6795 6449 6449
query15 249 172 173 172
query16 8087 245 241 241
query17 1914 509 489 489
query18 2120 274 254 254
query19 345 136 142 136
query20 80 77 74 74
query21 200 122 125 122
query22 4802 4594 4602 4594
query23 30746 29908 29979 29908
query24 9191 2759 2724 2724
query25 525 348 340 340
query26 707 141 147 141
query27 2169 290 294 290
query28 5765 1837 1815 1815
query29 861 602 595 595
query30 280 131 137 131
query31 915 706 708 706
query32 93 55 48 48
query33 534 221 212 212
query34 818 455 453 453
query35 862 737 745 737
query36 997 897 916 897
query37 93 56 54 54
query38 3205 3087 3100 3087
query39 1301 1244 1230 1230
query40 183 95 92 92
query41 39 35 34 34
query42 99 87 90 87
query43 520 463 478 463
query44 1011 661 679 661
query45 192 174 169 169
query46 1044 642 635 635
query47 1529 1414 1497 1414
query48 430 349 345 345
query49 1051 283 313 283
query50 752 364 367 364
query51 5251 5175 5163 5163
query52 103 81 81 81
query53 334 260 268 260
query54 247 219 215 215
query55 80 79 79 79
query56 221 198 193 193
query57 975 924 889 889
query58 196 168 176 168
query59 2485 2327 2295 2295
query60 240 217 209 209
query61 83 88 83 83
query62 627 373 361 361
query63 298 276 266 266
query64 4789 3712 3522 3522
query65 3230 3209 3209 3209
query66 814 306 302 302
query67 14563 14276 14241 14241
query68 4003 522 546 522
query69 461 349 315 315
query70 1188 1186 1179 1179
query71 328 235 243 235
query72 5946 2877 2681 2681
query73 694 314 318 314
query74 6693 6292 6231 6231
query75 3035 2378 2325 2325
query76 2552 950 965 950
query77 345 231 224 224
query78 9277 8848 8787 8787
query79 2606 490 487 487
query80 1801 351 349 349
query81 518 195 197 195
query82 673 79 78 78
query83 245 123 124 123
query84 274 88 77 77
query85 1779 337 332 332
query86 492 299 279 279
query87 3390 3208 3227 3208
query88 3872 2356 2338 2338
query89 431 365 356 356
query90 1942 169 162 162
query91 168 122 136 122
query92 59 40 42 40
query93 2541 476 488 476
query94 1576 191 189 189
query95 8735 382 8611 382
query96 612 287 273 273
query97 4314 4182 4225 4182
query98 469 236 331 236
query99 2473 703 694 694
Total cold run time: 286310 ms
Total hot run time: 173754 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1529609020
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSink.java:
##########
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Eliminate sort under sink, this can be used when sort is insensitive to the result
+ */
+public class EliminateSortUnderSink implements RewriteRuleFactory {
Review Comment:
have fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #30870:
URL: https://github.com/apache/doris/pull/30870#discussion_r1495334693
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/EliminateSortUnderSink.java:
##########
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.List;
+
+/**
+ * Eliminate sort under sink, this can be used when sort is insensitive to the result
+ */
+public class EliminateSortUnderSink implements RewriteRuleFactory {
+
+ @Override
+ public List<Rule> buildRules() {
+ return ImmutableList.of(
+ RuleType.ELIMINATE_SORT_UNDER_OLAP_TABLE_SINK.build(
+ logicalOlapTableSink(logicalSort((any()))).then(sink -> {
+ return sink.withChildren(sink.child().child());
+ })
+ )
+ );
+ }
Review Comment:
have not added it in rewritter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [test](neredis) Add tpch test for query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30870:
URL: https://github.com/apache/doris/pull/30870#issuecomment-1996748400
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org