You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "seawinde (via GitHub)" <gi...@apache.org> on 2024/09/13 08:21:53 UTC
[PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
seawinde opened a new pull request, #40803:
URL: https://github.com/apache/doris/pull/40803
## Proposed changes
This is brought by https://github.com/apache/doris/pull/36056
Not all query after rewritten successfully can compensate union all
Such as:
mv def sql is as following, partition column is a
```sql
select a, b, count(*) from t1 group by a, b
```
Query is as following:
```sq
select count(*) from t1
```
the result is
+----------+
| count(*) |
+----------+
| 24 |
+----------+
after rewritten by materialized view successfully
If mv part partition is invalid, can not compensate union all, because result is wrong after
compensate union all.
+----------+
| count(*) |
+----------+
| 24 |
| 3 |
+----------+
This pr fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2348446790
<details>
<summary>TPC-DS: <b>Total hot run time: 199281 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bbd2765c04f066e632ad9d4039715f7eda24f650, data reload: false
query1 1290 877 903 877
query2 6418 1812 1737 1737
query3 10656 3818 3870 3818
query4 55483 25956 24012 24012
query5 5030 531 522 522
query6 309 176 158 158
query7 5607 304 299 299
query8 275 226 233 226
query9 5998 2584 2599 2584
query10 427 294 277 277
query11 15978 15519 15711 15519
query12 151 101 100 100
query13 1409 401 382 382
query14 10576 7096 7096 7096
query15 197 174 171 171
query16 6825 508 478 478
query17 1160 611 611 611
query18 1876 299 331 299
query19 199 160 156 156
query20 118 116 118 116
query21 211 106 105 105
query22 4893 4542 4676 4542
query23 34664 33931 33527 33527
query24 6167 3118 3087 3087
query25 518 410 415 410
query26 604 162 159 159
query27 1558 284 291 284
query28 2828 2076 2063 2063
query29 676 424 434 424
query30 229 157 158 157
query31 993 777 824 777
query32 73 56 59 56
query33 481 320 312 312
query34 887 477 490 477
query35 836 732 746 732
query36 1027 885 915 885
query37 142 82 80 80
query38 3943 3949 3962 3949
query39 1453 1409 1392 1392
query40 211 115 120 115
query41 49 51 46 46
query42 120 102 99 99
query43 492 453 453 453
query44 1186 774 772 772
query45 199 170 170 170
query46 1107 835 824 824
query47 1880 1726 1804 1726
query48 368 300 293 293
query49 741 462 462 462
query50 927 446 442 442
query51 7260 7073 6949 6949
query52 97 88 91 88
query53 253 179 187 179
query54 573 452 474 452
query55 78 75 73 73
query56 273 263 266 263
query57 1202 1080 1065 1065
query58 225 232 244 232
query59 2670 2515 2502 2502
query60 290 267 270 267
query61 106 118 100 100
query62 814 670 658 658
query63 215 189 186 186
query64 1585 681 679 679
query65 3251 3158 3158 3158
query66 682 304 299 299
query67 15909 15735 15356 15356
query68 1585 868 856 856
query69 460 348 321 321
query70 1142 1107 1133 1107
query71 382 333 332 332
query72 6252 3449 3355 3355
query73 592 575 574 574
query74 9288 9204 9168 9168
query75 3115 2968 2972 2968
query76 1223 838 833 833
query77 510 388 397 388
query78 9361 9208 9265 9208
query79 913 891 871 871
query80 830 825 844 825
query81 462 268 261 261
query82 262 258 262 258
query83 203 195 189 189
query84 271 103 105 103
query85 676 414 397 397
query86 322 339 325 325
query87 4364 4358 4336 4336
query88 4122 4035 4042 4035
query89 376 363 365 363
query90 1733 305 302 302
query91 124 127 122 122
query92 80 76 80 76
query93 1046 1033 1034 1033
query94 624 380 386 380
query95 449 409 410 409
query96 473 466 466 466
query97 3105 3156 3142 3142
query98 226 226 238 226
query99 1630 1317 1288 1288
Total cold run time: 291129 ms
Total hot run time: 199281 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2350882402
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2360199721
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 merged PR #40803:
URL: https://github.com/apache/doris/pull/40803
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2357412799
<details>
<summary>ClickBench: <b>Total hot run time: 32.93 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd27e07daee20d8de3b609175b78d4944a1106c4, data reload: false
query1 0.04 0.05 0.04
query2 0.07 0.02 0.02
query3 0.22 0.06 0.06
query4 1.64 0.10 0.09
query5 0.53 0.50 0.52
query6 1.12 0.72 0.72
query7 0.02 0.02 0.02
query8 0.04 0.03 0.02
query9 0.55 0.52 0.49
query10 0.54 0.55 0.54
query11 0.13 0.10 0.11
query12 0.14 0.10 0.10
query13 0.60 0.59 0.58
query14 2.94 2.96 2.93
query15 0.88 0.82 0.83
query16 0.38 0.40 0.37
query17 1.07 1.05 1.02
query18 0.20 0.19 0.19
query19 1.91 1.90 2.03
query20 0.01 0.01 0.01
query21 15.35 0.61 0.61
query22 2.79 2.81 1.96
query23 17.14 0.90 0.83
query24 3.15 1.46 0.86
query25 0.26 0.14 0.15
query26 0.38 0.15 0.15
query27 0.04 0.04 0.03
query28 10.44 1.08 1.08
query29 12.58 3.25 3.25
query30 0.24 0.06 0.06
query31 2.86 0.39 0.37
query32 3.31 0.46 0.47
query33 2.99 3.05 3.06
query34 17.14 4.38 4.37
query35 4.39 4.48 4.43
query36 0.66 0.48 0.47
query37 0.08 0.07 0.05
query38 0.05 0.03 0.03
query39 0.03 0.02 0.02
query40 0.15 0.13 0.12
query41 0.07 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 107.2 s
Total hot run time: 32.93 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2359959963
<details>
<summary>TPC-DS: <b>Total hot run time: 194879 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 492ec6059c68359fed7b223af4548d3e446ce8a4, data reload: false
query1 972 368 364 364
query2 6511 2087 2051 2051
query3 6712 215 220 215
query4 37035 23344 23462 23344
query5 4889 469 468 468
query6 262 169 170 169
query7 4642 291 304 291
query8 280 252 221 221
query9 9396 2707 2693 2693
query10 489 285 295 285
query11 18064 15248 15166 15166
query12 162 95 95 95
query13 1630 415 388 388
query14 10846 7516 7401 7401
query15 293 181 173 173
query16 7982 459 446 446
query17 1706 584 554 554
query18 2084 311 314 311
query19 339 158 152 152
query20 116 106 106 106
query21 212 103 106 103
query22 4360 4110 4121 4110
query23 34524 33748 33923 33748
query24 11212 2980 2925 2925
query25 635 418 413 413
query26 1214 163 160 160
query27 2843 295 286 286
query28 8414 2493 2483 2483
query29 835 432 447 432
query30 320 162 156 156
query31 997 804 809 804
query32 89 58 59 58
query33 742 301 297 297
query34 904 492 494 492
query35 870 754 735 735
query36 1084 951 945 945
query37 162 89 86 86
query38 3974 3926 3923 3923
query39 1478 1424 1415 1415
query40 217 97 95 95
query41 52 48 51 48
query42 112 100 95 95
query43 538 503 502 502
query44 1245 817 807 807
query45 195 165 168 165
query46 1135 785 772 772
query47 1903 1793 1834 1793
query48 464 356 397 356
query49 1154 429 414 414
query50 828 404 410 404
query51 7143 6904 6814 6814
query52 103 91 86 86
query53 261 195 196 195
query54 1069 472 470 470
query55 84 83 84 83
query56 305 275 278 275
query57 1220 1106 1065 1065
query58 241 238 251 238
query59 3219 2982 2980 2980
query60 296 288 296 288
query61 126 127 130 127
query62 858 656 708 656
query63 217 188 194 188
query64 4973 680 645 645
query65 3275 3190 3178 3178
query66 1455 304 296 296
query67 15786 15519 15266 15266
query68 3118 870 865 865
query69 454 361 345 345
query70 1204 1171 1215 1171
query71 342 344 337 337
query72 5814 3338 3414 3338
query73 586 580 589 580
query74 9557 9024 9009 9009
query75 3055 2863 2857 2857
query76 1251 899 892 892
query77 413 382 380 380
query78 10374 9305 9264 9264
query79 912 898 895 895
query80 615 593 584 584
query81 451 256 252 252
query82 234 234 231 231
query83 159 159 162 159
query84 246 103 107 103
query85 674 385 361 361
query86 327 308 323 308
query87 4442 4348 4386 4348
query88 4837 4085 4069 4069
query89 375 362 363 362
query90 1410 323 320 320
query91 187 162 159 159
query92 82 75 73 73
query93 915 928 921 921
query94 565 391 369 369
query95 437 420 420 420
query96 503 487 492 487
query97 3115 3155 3131 3131
query98 235 235 229 229
query99 1416 1295 1306 1295
Total cold run time: 296058 ms
Total hot run time: 194879 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2348332913
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2357408412
<details>
<summary>TPC-DS: <b>Total hot run time: 199427 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd27e07daee20d8de3b609175b78d4944a1106c4, data reload: false
query1 1295 873 898 873
query2 6401 2058 2020 2020
query3 10820 4001 3932 3932
query4 63893 30126 23335 23335
query5 5200 493 467 467
query6 403 191 165 165
query7 5481 306 294 294
query8 320 221 225 221
query9 8580 2663 2636 2636
query10 455 279 277 277
query11 17945 15112 15845 15112
query12 157 105 101 101
query13 1472 409 408 408
query14 10796 7681 7313 7313
query15 206 188 174 174
query16 6788 518 512 512
query17 1138 627 602 602
query18 1117 302 306 302
query19 231 153 152 152
query20 127 114 110 110
query21 209 105 104 104
query22 4545 4503 4693 4503
query23 35509 34225 34023 34023
query24 6065 2928 2896 2896
query25 519 400 412 400
query26 647 159 162 159
query27 1701 280 297 280
query28 4001 2460 2400 2400
query29 666 429 426 426
query30 229 153 167 153
query31 974 788 793 788
query32 72 53 56 53
query33 397 285 295 285
query34 891 478 481 478
query35 867 736 718 718
query36 1057 945 929 929
query37 150 89 88 88
query38 4080 3929 4054 3929
query39 1468 1428 1371 1371
query40 205 96 96 96
query41 51 49 48 48
query42 116 97 98 97
query43 522 502 489 489
query44 1126 807 802 802
query45 199 164 168 164
query46 1111 772 760 760
query47 1936 1806 1811 1806
query48 465 360 367 360
query49 696 395 411 395
query50 915 393 411 393
query51 7061 6939 6898 6898
query52 101 88 86 86
query53 247 187 178 178
query54 556 446 462 446
query55 74 75 73 73
query56 272 279 252 252
query57 1205 1079 1117 1079
query58 211 225 236 225
query59 3206 3188 3140 3140
query60 297 269 259 259
query61 99 112 98 98
query62 748 675 665 665
query63 220 188 180 180
query64 1367 649 650 649
query65 3247 3178 3200 3178
query66 627 292 289 289
query67 15947 15735 15755 15735
query68 1272 852 839 839
query69 469 340 349 340
query70 1166 1110 1182 1110
query71 332 325 327 325
query72 6047 3594 3206 3206
query73 590 579 577 577
query74 9109 8934 9004 8934
query75 3026 2889 2923 2889
query76 1472 866 868 866
query77 496 368 368 368
query78 9487 9275 10613 9275
query79 891 876 876 876
query80 586 570 609 570
query81 461 251 249 249
query82 232 231 239 231
query83 155 155 152 152
query84 262 102 101 101
query85 690 371 353 353
query86 295 311 317 311
query87 4426 4339 4292 4292
query88 4457 4061 4047 4047
query89 363 358 358 358
query90 1778 309 314 309
query91 160 158 163 158
query92 71 72 72 72
query93 911 901 897 897
query94 565 370 349 349
query95 432 407 413 407
query96 487 487 485 485
query97 3179 3147 3134 3134
query98 223 220 220 220
query99 1403 1298 1299 1298
Total cold run time: 305192 ms
Total hot run time: 199427 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2359964993
<details>
<summary>ClickBench: <b>Total hot run time: 33.13 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 492ec6059c68359fed7b223af4548d3e446ce8a4, data reload: false
query1 0.05 0.05 0.04
query2 0.06 0.02 0.03
query3 0.23 0.06 0.07
query4 1.66 0.10 0.10
query5 0.52 0.49 0.49
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.55 0.49 0.49
query10 0.55 0.56 0.55
query11 0.14 0.11 0.12
query12 0.15 0.11 0.11
query13 0.61 0.60 0.59
query14 2.95 2.97 3.04
query15 0.88 0.83 0.81
query16 0.37 0.39 0.38
query17 1.01 1.02 1.08
query18 0.22 0.21 0.21
query19 1.81 1.90 1.93
query20 0.01 0.01 0.01
query21 15.36 0.59 0.57
query22 2.86 2.19 1.98
query23 17.25 0.74 0.84
query24 3.05 1.23 1.68
query25 0.38 0.13 0.11
query26 0.43 0.13 0.14
query27 0.05 0.04 0.04
query28 9.90 1.10 1.06
query29 12.56 3.22 3.21
query30 0.24 0.05 0.06
query31 2.88 0.40 0.37
query32 3.26 0.47 0.46
query33 2.97 3.06 3.05
query34 16.68 4.35 4.36
query35 4.42 4.39 4.43
query36 0.65 0.48 0.49
query37 0.08 0.06 0.06
query38 0.06 0.03 0.04
query39 0.03 0.02 0.02
query40 0.15 0.13 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.02 0.02
Total cold run time: 106.36 s
Total hot run time: 33.13 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2348459918
<details>
<summary>ClickBench: <b>Total hot run time: 30.96 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bbd2765c04f066e632ad9d4039715f7eda24f650, data reload: false
query1 0.04 0.04 0.05
query2 0.08 0.04 0.03
query3 0.22 0.04 0.04
query4 1.68 0.06 0.06
query5 0.51 0.49 0.50
query6 1.14 0.74 0.73
query7 0.02 0.02 0.01
query8 0.05 0.05 0.05
query9 0.57 0.52 0.51
query10 0.58 0.59 0.55
query11 0.16 0.12 0.12
query12 0.15 0.12 0.12
query13 0.63 0.61 0.60
query14 1.47 1.49 1.51
query15 0.92 0.89 0.89
query16 0.36 0.37 0.36
query17 1.02 1.03 1.03
query18 0.16 0.17 0.17
query19 1.88 1.80 1.80
query20 0.01 0.00 0.01
query21 15.39 0.70 0.68
query22 3.97 7.95 1.18
query23 17.95 1.33 1.30
query24 2.28 0.22 0.22
query25 0.17 0.08 0.08
query26 0.29 0.17 0.19
query27 0.08 0.08 0.08
query28 13.18 1.11 1.10
query29 12.51 3.36 3.31
query30 0.24 0.06 0.05
query31 2.85 0.41 0.41
query32 3.25 0.50 0.50
query33 3.06 3.06 3.05
query34 15.44 4.33 4.29
query35 4.37 4.34 4.36
query36 0.68 0.49 0.50
query37 0.18 0.16 0.16
query38 0.17 0.15 0.15
query39 0.05 0.04 0.04
query40 0.17 0.13 0.13
query41 0.10 0.05 0.06
query42 0.06 0.05 0.05
query43 0.05 0.04 0.05
Total cold run time: 108.14 s
Total hot run time: 30.96 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2357396729
<details>
<summary>TPC-H: <b>Total hot run time: 41301 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd27e07daee20d8de3b609175b78d4944a1106c4, data reload: false
------ Round 1 ----------------------------------
q1 17658 7460 7273 7273
q2 2037 154 164 154
q3 10583 1071 1147 1071
q4 10443 747 705 705
q5 7766 3067 3070 3067
q6 231 151 148 148
q7 1009 617 608 608
q8 9434 2054 2051 2051
q9 6779 6387 6394 6387
q10 7048 2255 2285 2255
q11 450 245 243 243
q12 403 212 217 212
q13 17797 3006 2959 2959
q14 239 211 209 209
q15 574 521 509 509
q16 691 602 625 602
q17 977 790 804 790
q18 7186 6662 6617 6617
q19 1402 1086 1071 1071
q20 581 287 281 281
q21 4020 3192 3077 3077
q22 1111 1031 1012 1012
Total cold run time: 108419 ms
Total hot run time: 41301 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7227 7227 7233 7227
q2 325 239 231 231
q3 2946 2925 3037 2925
q4 2002 1809 1816 1809
q5 5584 5620 5532 5532
q6 244 143 142 142
q7 2227 1813 1770 1770
q8 3284 3474 3399 3399
q9 8755 8906 8719 8719
q10 3558 3421 3462 3421
q11 581 488 478 478
q12 828 612 603 603
q13 10918 3169 3132 3132
q14 317 264 304 264
q15 585 524 531 524
q16 726 693 666 666
q17 1798 1610 1576 1576
q18 8158 7725 7861 7725
q19 1700 1685 1655 1655
q20 2110 1865 1867 1865
q21 5525 5231 5459 5231
q22 1140 1079 1053 1053
Total cold run time: 70538 ms
Total hot run time: 59947 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "morrySnow (via GitHub)" <gi...@apache.org>.
morrySnow commented on code in PR #40803:
URL: https://github.com/apache/doris/pull/40803#discussion_r1764522779
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/AbstractMaterializedViewAggregateRule.java:
##########
@@ -324,6 +326,46 @@ protected Expression tryRewriteExpression(StructInfo queryStructInfo, Expression
return rewrittenExpression;
}
+ /**
+ * Not all query after rewritten successfully can compensate union all
+ * Such as:
+ * mv def sql is as following, partition column is a
+ * select a, b, count(*) from t1 group by a, b
+ * Query is as following:
+ * select b, count(*) from t1 group by b, after rewritten by materialized view successfully
+ * If mv part partition is invalid, can not compensate union all, because result is wrong after
+ * compensate union all.
+ */
+ @Override
+ protected boolean canUnionRewrite(Plan queryPlan, MTMV mtmv, CascadesContext cascadesContext) {
+ // check query plan is contain the partition column
+ Optional<LogicalAggregate<Plan>> logicalAggregateOptional =
+ queryPlan.collectFirst(planTreeNode -> planTreeNode instanceof LogicalAggregate);
+ if (!logicalAggregateOptional.isPresent()) {
+ return true;
+ }
+
+ List<Expression> groupByExpressions = logicalAggregateOptional.get().getGroupByExpressions();
+ if (groupByExpressions.isEmpty()) {
+ // Scalar aggregate can not compensate union all
+ return false;
+ }
+ String relatedCol = mtmv.getMvPartitionInfo().getRelatedCol();
+ boolean canUnionRewrite = false;
+ // Check the query plan group by expression contains partition col or not
+ List<? extends Expression> groupByShuttledExpressions =
+ ExpressionUtils.shuttleExpressionWithLineage(groupByExpressions, queryPlan, new BitSet());
+ for (Expression expression : groupByShuttledExpressions) {
+ canUnionRewrite = !expression.collectToSet(expr -> expr instanceof SlotReference
+ && ((SlotReference) expr).isColumnFromTable()
+ && ((SlotReference) expr).getColumn().get().getName().equals(relatedCol)).isEmpty();
Review Comment:
check table name too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2359925481
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2359949077
<details>
<summary>TPC-H: <b>Total hot run time: 41754 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 492ec6059c68359fed7b223af4548d3e446ce8a4, data reload: false
------ Round 1 ----------------------------------
q1 17786 7444 7366 7366
q2 2435 177 168 168
q3 11089 1168 1230 1168
q4 10356 764 811 764
q5 7803 3127 3135 3127
q6 239 156 152 152
q7 1036 637 613 613
q8 9846 2068 2069 2068
q9 6850 6410 6388 6388
q10 7020 2256 2312 2256
q11 440 258 256 256
q12 411 228 221 221
q13 17789 2994 2961 2961
q14 249 208 204 204
q15 574 525 515 515
q16 682 626 610 610
q17 989 835 804 804
q18 7232 6720 6791 6720
q19 1416 982 997 982
q20 608 290 286 286
q21 4081 3388 3125 3125
q22 1099 1048 1000 1000
Total cold run time: 110030 ms
Total hot run time: 41754 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7268 7286 7251 7251
q2 326 233 228 228
q3 2889 2746 2771 2746
q4 1970 1711 1686 1686
q5 5381 5347 5392 5347
q6 223 145 144 144
q7 2067 1700 1714 1700
q8 3152 3317 3293 3293
q9 8409 8389 8403 8389
q10 3368 3334 3372 3334
q11 581 472 483 472
q12 789 605 602 602
q13 6425 2996 2970 2970
q14 288 259 286 259
q15 560 514 508 508
q16 711 670 678 670
q17 1744 1526 1532 1526
q18 7799 7360 7540 7360
q19 1679 1546 1620 1546
q20 2049 1819 1826 1819
q21 5353 5251 5245 5245
q22 1119 1013 1031 1013
Total cold run time: 64150 ms
Total hot run time: 58108 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2350911483
<details>
<summary>TPC-H: <b>Total hot run time: 43317 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd27e07daee20d8de3b609175b78d4944a1106c4, data reload: false
------ Round 1 ----------------------------------
q1 17600 7365 7327 7327
q2 2019 189 190 189
q3 10462 1293 1426 1293
q4 10313 1005 990 990
q5 7777 3273 3179 3179
q6 241 151 151 151
q7 1061 648 617 617
q8 9473 2084 2044 2044
q9 6871 6315 6383 6315
q10 7049 2571 2533 2533
q11 439 253 258 253
q12 411 241 232 232
q13 17764 3074 3069 3069
q14 285 255 265 255
q15 604 552 528 528
q16 527 448 445 445
q17 1033 968 960 960
q18 7492 7029 6824 6824
q19 1376 1266 1248 1248
q20 610 337 329 329
q21 3920 3591 3519 3519
q22 1096 1042 1017 1017
Total cold run time: 108423 ms
Total hot run time: 43317 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7172 7189 7239 7189
q2 360 238 243 238
q3 3137 3124 3099 3099
q4 2134 2146 2068 2068
q5 5741 5662 5796 5662
q6 243 148 149 148
q7 2153 1793 1804 1793
q8 3441 3557 3497 3497
q9 8989 8867 8856 8856
q10 3479 3622 3600 3600
q11 585 496 481 481
q12 823 643 654 643
q13 10234 3204 3205 3204
q14 320 299 293 293
q15 624 542 539 539
q16 514 471 475 471
q17 1826 1787 1779 1779
q18 8599 7982 8057 7982
q19 1805 1790 1763 1763
q20 2184 1883 1881 1881
q21 5950 5532 5673 5532
q22 1169 1071 1077 1071
Total cold run time: 71482 ms
Total hot run time: 61789 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2348332753
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2360199790
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](mtmv) Fix compensate union all wrongly when query rewrite by materialized view [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #40803:
URL: https://github.com/apache/doris/pull/40803#issuecomment-2348420346
<details>
<summary>TPC-H: <b>Total hot run time: 42642 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bbd2765c04f066e632ad9d4039715f7eda24f650, data reload: false
------ Round 1 ----------------------------------
q1 17618 7231 7181 7181
q2 2032 188 206 188
q3 10431 1278 1342 1278
q4 10490 1012 1067 1012
q5 7725 3169 3147 3147
q6 241 155 153 153
q7 1042 646 623 623
q8 9452 2027 1977 1977
q9 6739 6252 6286 6252
q10 7032 2513 2536 2513
q11 432 274 261 261
q12 399 231 229 229
q13 17758 3027 3034 3027
q14 295 246 249 246
q15 585 547 523 523
q16 517 428 423 423
q17 977 938 954 938
q18 7340 6875 6639 6639
q19 1381 1238 1220 1220
q20 598 346 342 342
q21 3883 3551 3486 3486
q22 1071 1009 984 984
Total cold run time: 108038 ms
Total hot run time: 42642 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7164 7123 7135 7123
q2 343 222 234 222
q3 3099 3102 3062 3062
q4 2088 2102 2068 2068
q5 5734 5613 5673 5613
q6 239 154 153 153
q7 2227 1802 1760 1760
q8 3377 3405 3422 3405
q9 8796 8953 8740 8740
q10 3604 3517 3562 3517
q11 579 487 486 486
q12 814 622 597 597
q13 9077 3200 3178 3178
q14 333 289 286 286
q15 582 529 541 529
q16 530 481 467 467
q17 1807 1759 1723 1723
q18 8437 7996 7995 7995
q19 1771 1743 1735 1735
q20 2127 1896 1864 1864
q21 5947 5626 5533 5533
q22 1121 1070 1046 1046
Total cold run time: 69796 ms
Total hot run time: 61102 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org