You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "seawinde (via GitHub)" <gi...@apache.org> on 2024/01/26 09:53:45 UTC
[PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
seawinde opened a new pull request, #30440:
URL: https://github.com/apache/doris/pull/30440
## Proposed changes
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914071675
<details>
<summary>ClickBench: <b>Total hot run time: 31.51 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1, data reload: false
query1 0.04 0.03 0.03
query2 0.06 0.02 0.03
query3 0.23 0.06 0.07
query4 1.65 0.11 0.10
query5 0.52 0.51 0.52
query6 1.19 0.63 0.64
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.53 0.51 0.49
query10 0.53 0.55 0.55
query11 0.12 0.08 0.08
query12 0.10 0.09 0.10
query13 0.61 0.60 0.60
query14 0.79 0.83 0.80
query15 0.79 0.78 0.77
query16 0.38 0.38 0.38
query17 1.03 0.98 0.96
query18 0.21 0.23 0.27
query19 1.86 1.76 1.78
query20 0.01 0.01 0.01
query21 15.42 0.59 0.58
query22 2.61 2.70 2.46
query23 17.25 0.84 0.86
query24 2.50 1.12 0.90
query25 0.29 0.29 0.09
query26 0.45 0.14 0.15
query27 0.05 0.05 0.04
query28 11.31 0.87 0.85
query29 12.57 3.17 3.25
query30 0.64 0.56 0.54
query31 2.79 0.34 0.35
query32 3.34 0.48 0.47
query33 3.22 3.25 3.20
query34 15.72 4.24 4.33
query35 4.25 4.24 4.28
query36 1.10 1.06 1.07
query37 0.06 0.05 0.05
query38 0.04 0.03 0.02
query39 0.03 0.01 0.02
query40 0.16 0.12 0.13
query41 0.07 0.01 0.02
query42 0.02 0.01 0.02
query43 0.03 0.02 0.02
Total cold run time: 104.63 s
Total hot run time: 31.51 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913526189
<details>
<summary>TPC-H: <b>Total hot run time: 37849 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7, data reload: false
------ Round 1 ----------------------------------
q1 17883 5220 5015 5015
q2 2877 144 138 138
q3 11537 939 956 939
q4 4957 828 757 757
q5 7902 2871 2895 2871
q6 192 125 124 124
q7 1175 756 759 756
q8 9488 2032 2061 2032
q9 7200 6354 6346 6346
q10 8106 2430 2415 2415
q11 415 202 210 202
q12 770 269 283 269
q13 18024 3310 3326 3310
q14 270 254 253 253
q15 542 492 483 483
q16 460 399 393 393
q17 946 543 538 538
q18 7014 5964 6196 5964
q19 1581 1445 1307 1307
q20 602 328 333 328
q21 6797 3127 3117 3117
q22 824 292 292 292
Total cold run time: 109562 ms
Total hot run time: 37849 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4986 5180 4985 4985
q2 343 255 253 253
q3 3038 2691 2712 2691
q4 1861 1627 1556 1556
q5 5460 5252 5196 5196
q6 207 116 119 116
q7 2181 1735 1845 1735
q8 3122 3212 3246 3212
q9 8686 8521 11385 8521
q10 5421 3475 3484 3475
q11 549 480 459 459
q12 761 593 610 593
q13 16963 3135 3052 3052
q14 275 250 266 250
q15 531 491 484 484
q16 523 483 468 468
q17 2100 1627 1689 1627
q18 9007 15413 8954 8954
q19 26879 1594 1511 1511
q20 4640 1903 1905 1903
q21 16061 4820 4824 4820
q22 1537 461 488 461
Total cold run time: 115131 ms
Total hot run time: 56322 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914023689
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913893098
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #30440:
URL: https://github.com/apache/doris/pull/30440#discussion_r1469041729
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewAggregateRule.java:
##########
@@ -438,9 +439,110 @@ private boolean isAggregateFunctionEquivalent(Function queryFunction, Function v
* actualFunction is bitmap_union(to_bitmap(case when a = 5 then 1 else 2 end))
* after extracting, the return argument is: case when a = 5 then 1 else 2 end
*/
- private List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
+ private static List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
Set<Object> exprSetToRemove = functionWithAny.collectToSet(expr -> !(expr instanceof Any));
return actualFunction.collectFirst(expr ->
exprSetToRemove.stream().noneMatch(exprToRemove -> exprToRemove.equals(expr)));
}
+
+ /**
+ * Aggregate expression rewriter which is responsible for rewriting group by and
+ * aggregate function expression
+ */
+ protected static class AggregateExpressionRewriter
+ extends DefaultExpressionRewriter<AggregateExpressionRewriteContext> {
+
+ @Override
+ public Expression visitAggregateFunction(AggregateFunction aggregateFunction,
+ AggregateExpressionRewriteContext rewriteContext) {
+ if (!rewriteContext.isValid()) {
+ return null;
+ }
+ Expression queryFunctionShuttled = ExpressionUtils.shuttleExpressionWithLineage(
+ aggregateFunction,
+ rewriteContext.getQueryTopPlan());
+ Function rollupAggregateFunction = rollup(aggregateFunction, queryFunctionShuttled,
+ rewriteContext.getMvExprToMvScanExprQueryBased());
+ if (rollupAggregateFunction == null) {
+ rewriteContext.setValid(false);
+ return null;
+ }
+ return rollupAggregateFunction;
+ }
+
+ @Override
+ public Expression visitSlot(Slot slot, AggregateExpressionRewriteContext rewriteContext) {
+ if (!rewriteContext.isValid()) {
+ return null;
+ }
+ if (rewriteContext.getMvExprToMvScanExprQueryBased().containsKey(slot)) {
+ return rewriteContext.getMvExprToMvScanExprQueryBased().get(slot);
+ }
+ rewriteContext.setValid(false);
+ return null;
+ }
Review Comment:
```suggestion
return originExpr;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913893072
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914065730
<details>
<summary>TPC-DS: <b>Total hot run time: 172972 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1, data reload: false
query1 950 334 323 323
query2 6585 2062 1844 1844
query3 6699 198 196 196
query4 33409 21979 22111 21979
query5 4448 419 348 348
query6 244 151 148 148
query7 4605 261 265 261
query8 244 168 168 168
query9 8571 2264 2241 2241
query10 409 202 207 202
query11 17805 15529 15288 15288
query12 114 66 67 66
query13 1675 365 373 365
query14 9123 6543 6522 6522
query15 210 179 177 177
query16 5701 256 255 255
query17 946 475 468 468
query18 1804 256 250 250
query19 181 134 135 134
query20 74 63 63 63
query21 205 135 129 129
query22 4866 4712 4742 4712
query23 31127 30444 30433 30433
query24 11619 2763 2805 2763
query25 590 334 300 300
query26 1818 136 135 135
query27 3201 283 280 280
query28 7555 1838 1813 1813
query29 1867 611 590 590
query30 282 138 136 136
query31 890 715 709 709
query32 77 52 48 48
query33 702 215 201 201
query34 1160 451 460 451
query35 923 754 739 739
query36 1296 1224 1214 1214
query37 91 54 57 54
query38 3252 3186 3168 3168
query39 1290 1252 1243 1243
query40 339 87 84 84
query41 38 38 34 34
query42 91 80 82 80
query43 528 473 500 473
query44 1133 686 702 686
query45 193 175 170 170
query46 1070 654 653 653
query47 1604 1498 1518 1498
query48 386 312 320 312
query49 1238 282 277 277
query50 687 311 313 311
query51 5348 5188 5209 5188
query52 97 83 78 78
query53 316 257 262 257
query54 253 184 184 184
query55 83 69 72 69
query56 185 156 157 156
query57 963 887 904 887
query58 190 159 154 154
query59 2599 2587 2380 2380
query60 198 177 177 177
query61 85 86 80 80
query62 605 355 363 355
query63 286 274 255 255
query64 5970 1743 1721 1721
query65 3271 3230 3223 3223
query66 1240 323 309 309
query67 14528 14195 14089 14089
query68 4041 496 476 476
query69 420 297 302 297
query70 1506 1482 1495 1482
query71 253 230 211 211
query72 3799 2846 2838 2838
query73 688 321 314 314
query74 6764 6209 6303 6209
query75 2858 2321 2308 2308
query76 2587 1069 1046 1046
query77 368 227 228 227
query78 9124 8770 8589 8589
query79 2009 490 488 488
query80 1373 320 307 307
query81 518 197 201 197
query82 678 78 77 77
query83 229 117 110 110
query84 276 71 78 71
query85 2119 335 326 326
query86 521 404 401 401
query87 3434 3276 3303 3276
query88 3913 2184 2178 2178
query89 415 360 365 360
query90 1875 188 188 188
query91 152 114 120 114
query92 55 42 43 42
query93 2112 420 436 420
query94 1355 158 160 158
query95 497 473 437 437
query96 619 314 315 314
query97 4275 4118 4127 4118
query98 202 198 184 184
query99 1074 671 653 653
Total cold run time: 280241 ms
Total hot run time: 172972 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1911776287
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30440:
URL: https://github.com/apache/doris/pull/30440#discussion_r1469133541
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewAggregateRule.java:
##########
@@ -438,9 +439,110 @@ private boolean isAggregateFunctionEquivalent(Function queryFunction, Function v
* actualFunction is bitmap_union(to_bitmap(case when a = 5 then 1 else 2 end))
* after extracting, the return argument is: case when a = 5 then 1 else 2 end
*/
- private List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
+ private static List<Expression> extractArguments(Expression functionWithAny, Function actualFunction) {
Set<Object> exprSetToRemove = functionWithAny.collectToSet(expr -> !(expr instanceof Any));
return actualFunction.collectFirst(expr ->
exprSetToRemove.stream().noneMatch(exprToRemove -> exprToRemove.equals(expr)));
}
+
+ /**
+ * Aggregate expression rewriter which is responsible for rewriting group by and
+ * aggregate function expression
+ */
+ protected static class AggregateExpressionRewriter
+ extends DefaultExpressionRewriter<AggregateExpressionRewriteContext> {
+
+ @Override
+ public Expression visitAggregateFunction(AggregateFunction aggregateFunction,
+ AggregateExpressionRewriteContext rewriteContext) {
+ if (!rewriteContext.isValid()) {
+ return null;
+ }
+ Expression queryFunctionShuttled = ExpressionUtils.shuttleExpressionWithLineage(
+ aggregateFunction,
+ rewriteContext.getQueryTopPlan());
+ Function rollupAggregateFunction = rollup(aggregateFunction, queryFunctionShuttled,
+ rewriteContext.getMvExprToMvScanExprQueryBased());
+ if (rollupAggregateFunction == null) {
+ rewriteContext.setValid(false);
+ return null;
+ }
+ return rollupAggregateFunction;
+ }
+
+ @Override
+ public Expression visitSlot(Slot slot, AggregateExpressionRewriteContext rewriteContext) {
+ if (!rewriteContext.isValid()) {
+ return null;
+ }
+ if (rewriteContext.getMvExprToMvScanExprQueryBased().containsKey(slot)) {
+ return rewriteContext.getMvExprToMvScanExprQueryBased().get(slot);
+ }
+ rewriteContext.setValid(false);
+ return null;
+ }
Review Comment:
Have fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913529800
<details>
<summary>TPC-DS: <b>Total hot run time: 172622 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7, data reload: false
query1 926 335 330 330
query2 6545 1897 2048 1897
query3 6709 200 199 199
query4 32579 22154 22121 22121
query5 4447 410 342 342
query6 254 159 156 156
query7 4600 263 257 257
query8 240 180 170 170
query9 8582 2269 2244 2244
query10 422 214 199 199
query11 17874 15516 15309 15309
query12 110 66 64 64
query13 1692 379 378 378
query14 9202 6378 6479 6378
query15 217 185 182 182
query16 5711 259 243 243
query17 951 479 463 463
query18 1795 248 253 248
query19 169 134 130 130
query20 75 64 68 64
query21 203 134 135 134
query22 4678 4881 4516 4516
query23 31244 30397 30309 30309
query24 12421 2752 2756 2752
query25 599 313 307 307
query26 1973 144 137 137
query27 3181 284 282 282
query28 7741 1820 1810 1810
query29 2248 598 608 598
query30 290 135 134 134
query31 892 689 721 689
query32 75 47 50 47
query33 712 213 210 210
query34 1163 450 471 450
query35 849 756 775 756
query36 1319 1139 1120 1120
query37 98 59 60 59
query38 3243 3161 3120 3120
query39 1327 1252 1256 1252
query40 342 89 82 82
query41 38 35 34 34
query42 89 79 83 79
query43 546 501 505 501
query44 1101 679 686 679
query45 191 175 172 172
query46 1033 659 653 653
query47 1621 1446 1547 1446
query48 376 326 300 300
query49 1225 283 286 283
query50 668 315 316 315
query51 5259 5124 5180 5124
query52 85 78 80 78
query53 323 257 265 257
query54 238 186 186 186
query55 76 77 74 74
query56 174 155 167 155
query57 1039 914 903 903
query58 185 150 152 150
query59 2800 2733 2525 2525
query60 202 176 179 176
query61 81 84 81 81
query62 651 365 367 365
query63 279 268 268 268
query64 5985 1752 1743 1743
query65 3300 3225 3224 3224
query66 1457 316 313 313
query67 14095 14383 14221 14221
query68 3741 495 498 495
query69 417 305 295 295
query70 1539 1456 1493 1456
query71 292 208 240 208
query72 3671 2836 2847 2836
query73 699 316 322 316
query74 6666 6192 6289 6192
query75 2843 2314 2310 2310
query76 2610 1015 1026 1015
query77 427 235 228 228
query78 9006 8705 8501 8501
query79 2117 482 500 482
query80 1383 322 329 322
query81 522 202 200 200
query82 680 84 78 78
query83 231 113 123 113
query84 279 79 73 73
query85 2201 352 323 323
query86 500 399 380 380
query87 3374 3256 3284 3256
query88 3872 2150 2138 2138
query89 421 354 359 354
query90 2052 186 181 181
query91 151 116 119 116
query92 56 43 44 43
query93 2125 440 409 409
query94 1471 162 159 159
query95 496 455 463 455
query96 626 304 316 304
query97 4243 4090 4101 4090
query98 206 204 192 192
query99 1049 659 688 659
Total cold run time: 280781 ms
Total hot run time: 172622 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914054541
<details>
<summary>TPC-H: <b>Total hot run time: 37540 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1, data reload: false
------ Round 1 ----------------------------------
q1 17647 5306 4936 4936
q2 2044 143 131 131
q3 10696 902 929 902
q4 4699 679 724 679
q5 7664 2878 2822 2822
q6 185 117 118 117
q7 1123 736 711 711
q8 9339 1991 2024 1991
q9 7262 6295 6283 6283
q10 8101 2409 2428 2409
q11 403 207 202 202
q12 765 284 271 271
q13 18004 3273 3305 3273
q14 278 257 242 242
q15 524 488 483 483
q16 452 400 401 400
q17 932 572 564 564
q18 7150 6065 6038 6038
q19 1576 1395 1389 1389
q20 577 328 352 328
q21 6951 3083 3113 3083
q22 803 286 290 286
Total cold run time: 107175 ms
Total hot run time: 37540 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5175 5000 5022 5000
q2 352 248 238 238
q3 2993 2662 2690 2662
q4 1833 1512 1576 1512
q5 5271 5163 5243 5163
q6 202 113 116 113
q7 2112 1761 1737 1737
q8 3160 3159 3185 3159
q9 8608 8635 10620 8635
q10 5480 3436 3468 3436
q11 546 466 465 465
q12 765 642 583 583
q13 16915 3112 3065 3065
q14 283 250 265 250
q15 531 483 493 483
q16 519 475 464 464
q17 2019 1674 1570 1570
q18 8868 10215 8562 8562
q19 24259 1604 1539 1539
q20 4627 1897 1878 1878
q21 15820 4840 4792 4792
q22 1113 472 502 472
Total cold run time: 111451 ms
Total hot run time: 55778 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914030662
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow merged PR #30440:
URL: https://github.com/apache/doris/pull/30440
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913531205
<details>
<summary>ClickBench: <b>Total hot run time: 30.96 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7, data reload: false
query1 0.03 0.04 0.03
query2 0.06 0.02 0.03
query3 0.23 0.06 0.06
query4 1.66 0.11 0.10
query5 0.52 0.52 0.50
query6 1.19 0.62 0.61
query7 0.02 0.02 0.01
query8 0.04 0.02 0.03
query9 0.55 0.49 0.50
query10 0.57 0.55 0.56
query11 0.12 0.09 0.08
query12 0.11 0.09 0.10
query13 0.59 0.60 0.60
query14 0.79 0.79 0.81
query15 0.79 0.78 0.78
query16 0.38 0.39 0.37
query17 1.00 1.03 1.03
query18 0.25 0.24 0.27
query19 1.89 1.79 1.79
query20 0.01 0.01 0.01
query21 15.41 0.55 0.57
query22 2.66 2.41 2.09
query23 17.29 0.91 0.72
query24 2.42 1.65 0.45
query25 0.32 0.25 0.23
query26 0.49 0.13 0.14
query27 0.06 0.04 0.04
query28 11.40 0.84 0.86
query29 12.53 3.20 3.21
query30 0.67 0.55 0.60
query31 2.79 0.34 0.36
query32 3.36 0.48 0.48
query33 3.20 3.22 3.24
query34 16.00 4.29 4.32
query35 4.44 4.36 4.37
query36 1.09 1.05 1.06
query37 0.07 0.05 0.05
query38 0.04 0.03 0.02
query39 0.02 0.02 0.02
query40 0.17 0.13 0.12
query41 0.07 0.01 0.01
query42 0.02 0.01 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.35 s
Total hot run time: 30.96 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913532105
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 019c33c82e8e7b7931328b89e06fbb8bdd8559d7 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.7 seconds inserted 10000000 Rows, about 729K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1914075115
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit d90e1c0495d37fbb7a923da38d25246b3963c7b1 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select: 13.8 seconds inserted 10000000 Rows, about 724K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](Nereids) Support aggregate rewrite by materialized view with complex expression [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #30440:
URL: https://github.com/apache/doris/pull/30440#issuecomment-1913519851
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org