You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "starocean999 (via GitHub)" <gi...@apache.org> on 2024/06/05 04:18:45 UTC
[PR] [fix](nereids)disable unique key compute for mv [doris]
starocean999 opened a new pull request, #35898:
URL: https://github.com/apache/doris/pull/35898
## Proposed changes
computing unique doesn't work for mv, because mv's key may be different from base table and the key can be any expression which is difficult to deduce if it's unique. for example
base table:
```
CREATE TABLE IF NOT EXISTS base(
siteid INT(11) NOT NULL,
citycode SMALLINT(6) NOT NULL,
username VARCHAR(32) NOT NULL,
pv BIGINT(20) SUM NOT NULL DEFAULT '0'
)
AGGREGATE KEY (siteid,citycode,username)
DISTRIBUTED BY HASH(siteid) BUCKETS 5 properties("replication_num" = "1");
```
case1:
create mv1:
`create materialized view mv1 as select siteid, sum(pv) from base group by siteid;`
the base table siteid + citycode + username is unique but the mv1's agg key siteid is not unique
case2:
create mv2:
`create materialized view mv2 as select citycode * citycode, siteid, username from base;`
the mv2's agg key citycode * citycode is not unique
for simplicity, we disable unique compute for mv
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2151328964
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628714500
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java:
##########
@@ -128,20 +127,6 @@ public String qualifiedName() {
@Override
public void computeUnique(DataTrait.Builder builder) {
Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
- if (table instanceof OlapTable && ((OlapTable) table).getKeysType().isAggregationFamily()) {
- ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
- for (Slot slot : outputSet) {
- if (!(slot instanceof SlotReference)) {
- continue;
- }
- SlotReference slotRef = (SlotReference) slot;
- if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
- uniqSlots.add(slot);
- }
- }
- builder.addUniqueSlot(uniqSlots.build());
- }
-
Review Comment:
Add a condition to check if it is a mv rather than remove those codes. They are useful if it's a trivial olapscan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 merged PR #35898:
URL: https://github.com/apache/doris/pull/35898
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172849013
<details>
<summary>TPC-DS: <b>Total hot run time: 172208 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94f4de5c91b98645a878ff189333f169da09df94, data reload: false
query1 922 389 373 373
query2 6473 2424 2309 2309
query3 6648 209 212 209
query4 19461 17112 17271 17112
query5 4148 482 467 467
query6 269 167 172 167
query7 4595 304 294 294
query8 314 293 283 283
query9 8539 2421 2388 2388
query10 610 305 290 290
query11 10615 10043 9961 9961
query12 140 91 84 84
query13 1656 390 367 367
query14 10128 6988 7043 6988
query15 227 192 194 192
query16 7740 259 260 259
query17 1843 539 568 539
query18 1942 283 273 273
query19 188 153 156 153
query20 93 88 88 88
query21 211 134 130 130
query22 4279 4030 3994 3994
query23 33524 32972 33004 32972
query24 12069 2914 2837 2837
query25 652 353 369 353
query26 1810 161 154 154
query27 2929 326 326 326
query28 7195 2064 2031 2031
query29 1100 634 623 623
query30 284 153 155 153
query31 936 737 735 735
query32 86 55 56 55
query33 772 305 295 295
query34 928 511 476 476
query35 740 624 605 605
query36 1063 937 956 937
query37 292 70 72 70
query38 2867 2730 2700 2700
query39 863 793 770 770
query40 289 127 127 127
query41 58 56 56 56
query42 126 98 107 98
query43 607 536 542 536
query44 1218 729 729 729
query45 198 164 165 164
query46 1076 743 726 726
query47 1838 1789 1755 1755
query48 383 311 301 301
query49 1214 404 402 402
query50 769 401 397 397
query51 6913 6757 6583 6583
query52 102 95 92 92
query53 364 288 291 288
query54 989 463 445 445
query55 77 76 75 75
query56 286 248 251 248
query57 1170 1048 1069 1048
query58 254 282 240 240
query59 3397 3260 3058 3058
query60 293 278 284 278
query61 97 90 92 90
query62 657 445 444 444
query63 319 296 290 290
query64 9928 2300 1740 1740
query65 3234 3087 3129 3087
query66 1362 343 340 340
query67 15500 14938 14771 14771
query68 4576 543 544 543
query69 463 318 309 309
query70 1120 1144 1116 1116
query71 399 282 269 269
query72 7196 5350 5545 5350
query73 751 322 327 322
query74 5881 5469 5512 5469
query75 3365 2732 2689 2689
query76 2761 929 929 929
query77 444 315 294 294
query78 10197 9847 9821 9821
query79 2281 525 525 525
query80 830 473 461 461
query81 567 233 225 225
query82 768 106 101 101
query83 284 179 182 179
query84 241 93 87 87
query85 1264 284 268 268
query86 458 320 338 320
query87 3278 3079 3121 3079
query88 4183 2406 2377 2377
query89 479 390 400 390
query90 1782 264 197 197
query91 133 98 99 98
query92 67 50 47 47
query93 1781 515 502 502
query94 1259 190 191 190
query95 398 311 320 311
query96 599 271 271 271
query97 3211 3047 3028 3028
query98 215 202 189 189
query99 1149 836 832 832
Total cold run time: 274249 ms
Total hot run time: 172208 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172859510
<details>
<summary>ClickBench: <b>Total hot run time: 30.5 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94f4de5c91b98645a878ff189333f169da09df94, data reload: false
query1 0.04 0.03 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.73 0.08 0.10
query5 0.49 0.48 0.48
query6 1.14 0.73 0.73
query7 0.02 0.01 0.02
query8 0.05 0.05 0.05
query9 0.53 0.49 0.49
query10 0.56 0.55 0.53
query11 0.16 0.11 0.12
query12 0.14 0.11 0.12
query13 0.60 0.59 0.60
query14 0.79 0.77 0.77
query15 0.86 0.81 0.80
query16 0.38 0.35 0.35
query17 0.99 0.97 0.97
query18 0.24 0.25 0.27
query19 1.82 1.70 1.68
query20 0.01 0.01 0.01
query21 15.41 0.64 0.64
query22 4.37 7.01 1.94
query23 18.28 1.42 1.28
query24 2.09 0.22 0.21
query25 0.15 0.08 0.08
query26 0.26 0.18 0.17
query27 0.09 0.08 0.07
query28 13.26 1.03 0.99
query29 12.63 3.38 3.32
query30 0.26 0.06 0.06
query31 2.86 0.39 0.39
query32 3.28 0.48 0.47
query33 2.90 2.92 2.87
query34 17.07 4.39 4.38
query35 4.51 4.56 4.44
query36 0.65 0.46 0.47
query37 0.18 0.16 0.16
query38 0.16 0.15 0.15
query39 0.04 0.03 0.04
query40 0.17 0.16 0.14
query41 0.10 0.04 0.04
query42 0.05 0.04 0.05
query43 0.04 0.04 0.04
Total cold run time: 109.67 s
Total hot run time: 30.5 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148823650
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148885765
<details>
<summary>TPC-DS: <b>Total hot run time: 169554 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c1ea4dcd61a2edb41cb8b87b7158ec4ce59929a, data reload: false
query1 942 378 368 368
query2 6460 2416 2299 2299
query3 6651 208 209 208
query4 20038 17184 17535 17184
query5 4164 453 453 453
query6 260 164 164 164
query7 4593 306 283 283
query8 315 279 278 278
query9 8550 2419 2392 2392
query10 463 316 286 286
query11 10545 10072 9991 9991
query12 137 87 85 85
query13 1639 356 364 356
query14 9211 7597 6790 6790
query15 229 187 184 184
query16 7823 259 263 259
query17 1553 521 514 514
query18 1945 284 281 281
query19 193 151 148 148
query20 91 87 91 87
query21 205 136 131 131
query22 4430 4212 3958 3958
query23 33744 33060 33192 33060
query24 12176 2810 2809 2809
query25 661 352 358 352
query26 1829 155 161 155
query27 3066 317 321 317
query28 7702 2048 2043 2043
query29 1106 613 603 603
query30 285 171 149 149
query31 953 735 742 735
query32 85 53 54 53
query33 747 274 267 267
query34 1014 482 475 475
query35 753 615 629 615
query36 1117 939 892 892
query37 183 69 69 69
query38 2909 2771 2756 2756
query39 861 793 787 787
query40 282 122 124 122
query41 49 50 51 50
query42 115 95 100 95
query43 600 537 542 537
query44 1232 727 752 727
query45 207 173 165 165
query46 1073 732 744 732
query47 1866 1778 1791 1778
query48 386 287 300 287
query49 1180 407 443 407
query50 779 391 394 391
query51 6715 6681 6664 6664
query52 101 104 87 87
query53 352 291 289 289
query54 984 439 433 433
query55 76 75 73 73
query56 271 250 253 250
query57 1154 1035 1091 1035
query58 252 265 239 239
query59 3380 3224 3269 3224
query60 294 264 269 264
query61 91 86 118 86
query62 647 448 456 448
query63 317 296 284 284
query64 9839 2231 1712 1712
query65 3149 3094 3084 3084
query66 1371 340 326 326
query67 15442 14851 14895 14851
query68 4539 544 533 533
query69 465 309 308 308
query70 1181 1126 1156 1126
query71 394 281 289 281
query72 7181 5886 2904 2904
query73 742 329 321 321
query74 5951 5587 5435 5435
query75 3378 2649 2699 2649
query76 2439 976 890 890
query77 462 306 298 298
query78 10447 9843 9651 9651
query79 2941 512 509 509
query80 1961 492 468 468
query81 585 224 223 223
query82 717 103 106 103
query83 342 180 177 177
query84 276 89 92 89
query85 1889 330 325 325
query86 482 323 311 311
query87 3273 3069 3120 3069
query88 3950 2358 2368 2358
query89 499 397 386 386
query90 1845 194 204 194
query91 137 108 109 108
query92 70 52 53 52
query93 4447 521 501 501
query94 1290 196 200 196
query95 418 322 326 322
query96 604 266 263 263
query97 3227 3029 2993 2993
query98 239 222 231 222
query99 1375 813 819 813
Total cold run time: 279578 ms
Total hot run time: 169554 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148892309
<details>
<summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c1ea4dcd61a2edb41cb8b87b7158ec4ce59929a, data reload: false
query1 0.04 0.04 0.04
query2 0.08 0.04 0.04
query3 0.24 0.04 0.05
query4 1.68 0.09 0.09
query5 0.50 0.49 0.49
query6 1.12 0.72 0.73
query7 0.03 0.01 0.02
query8 0.06 0.04 0.04
query9 0.55 0.47 0.50
query10 0.55 0.55 0.54
query11 0.16 0.11 0.11
query12 0.14 0.11 0.12
query13 0.60 0.59 0.60
query14 0.76 0.78 0.77
query15 0.84 0.81 0.80
query16 0.36 0.36 0.38
query17 0.99 0.96 1.03
query18 0.22 0.26 0.25
query19 1.84 1.81 1.80
query20 0.02 0.01 0.02
query21 15.73 0.65 0.66
query22 4.35 7.42 2.04
query23 18.22 1.34 1.20
query24 2.06 0.19 0.20
query25 0.13 0.07 0.08
query26 0.28 0.18 0.18
query27 0.07 0.07 0.08
query28 13.56 1.02 0.99
query29 13.49 3.25 3.26
query30 0.24 0.06 0.06
query31 2.87 0.39 0.39
query32 3.25 0.48 0.46
query33 2.85 2.90 2.98
query34 17.27 4.40 4.37
query35 4.50 4.48 4.48
query36 0.65 0.48 0.45
query37 0.17 0.14 0.14
query38 0.15 0.14 0.14
query39 0.04 0.03 0.04
query40 0.17 0.15 0.15
query41 0.09 0.05 0.04
query42 0.06 0.05 0.04
query43 0.04 0.04 0.03
Total cold run time: 111.02 s
Total hot run time: 30.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "starocean999 (via GitHub)" <gi...@apache.org>.
starocean999 commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172593878
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148823602
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2148873802
<details>
<summary>TPC-H: <b>Total hot run time: 41019 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c1ea4dcd61a2edb41cb8b87b7158ec4ce59929a, data reload: false
------ Round 1 ----------------------------------
q1 17735 4399 4295 4295
q2 2034 189 195 189
q3 10619 1234 1182 1182
q4 10623 933 785 785
q5 7528 2766 2770 2766
q6 227 134 136 134
q7 965 610 610 610
q8 9257 2121 2091 2091
q9 9656 6649 6633 6633
q10 9726 3815 3723 3723
q11 453 246 245 245
q12 473 232 225 225
q13 19032 2917 2997 2917
q14 262 205 226 205
q15 511 483 467 467
q16 525 374 383 374
q17 1000 722 718 718
q18 8060 7424 7528 7424
q19 3254 1495 1480 1480
q20 661 314 309 309
q21 4902 3939 3911 3911
q22 404 336 337 336
Total cold run time: 117907 ms
Total hot run time: 41019 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4412 4202 4245 4202
q2 385 276 265 265
q3 2979 2768 2779 2768
q4 1881 1613 1622 1613
q5 5260 5269 5305 5269
q6 213 123 124 123
q7 2139 1784 1774 1774
q8 3200 3395 3307 3307
q9 8365 8395 8320 8320
q10 3893 3621 3667 3621
q11 582 481 488 481
q12 786 590 604 590
q13 16422 2987 2983 2983
q14 290 277 260 260
q15 516 481 482 481
q16 461 401 428 401
q17 1786 1471 1472 1471
q18 7668 7734 7466 7466
q19 1750 1621 1651 1621
q20 1997 1787 1809 1787
q21 4935 4673 4809 4673
q22 605 525 545 525
Total cold run time: 70525 ms
Total hot run time: 54001 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628717517
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java:
##########
@@ -455,4 +457,51 @@ public JSONObject toJson() {
olapScan.put("Properties", properties);
return olapScan;
}
+
+ @Override
+ public void computeUnique(DataTrait.Builder builder) {
+ super.computeUnique(builder);
+ if (this.selectedIndexId != getTable().getBaseIndexId()) {
+ /*
+ computing unique doesn't work for mv, because mv's key may be different from base table
+ and the key can be any expression which is difficult to deduce if it's unique. for example
+ base table:
+ CREATE TABLE IF NOT EXISTS base(
+ siteid INT(11) NOT NULL,
+ citycode SMALLINT(6) NOT NULL,
+ username VARCHAR(32) NOT NULL,
+ pv BIGINT(20) SUM NOT NULL DEFAULT '0'
+ )
+ AGGREGATE KEY (siteid,citycode,username)
+ DISTRIBUTED BY HASH(siteid) BUCKETS 5 properties("replication_num" = "1");
+
+ case1:
+ create mv1:
+ create materialized view mv1 as select siteid, sum(pv) from base group by siteid;
+ the base table siteid + citycode + username is unique but the mv1's agg key siteid is not unique
+
+ case2:
+ create mv2:
+ create materialized view mv2 as select citycode * citycode, siteid, username from base;
+ the mv2's agg key citycode * citycode is not unique
+
+ for simplicity, we disable unique compute for mv
+ */
+ return;
+ }
+ Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
+ if (getTable().getKeysType().isAggregationFamily()) {
+ ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
+ for (Slot slot : outputSet) {
+ if (!(slot instanceof SlotReference)) {
+ continue;
+ }
+ SlotReference slotRef = (SlotReference) slot;
+ if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
+ uniqSlots.add(slot);
+ }
+ }
+ builder.addUniqueSlot(uniqSlots.build());
+ }
+ }
Review Comment:
You should add the constraint code here, too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628714500
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalCatalogRelation.java:
##########
@@ -128,20 +127,6 @@ public String qualifiedName() {
@Override
public void computeUnique(DataTrait.Builder builder) {
Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
- if (table instanceof OlapTable && ((OlapTable) table).getKeysType().isAggregationFamily()) {
- ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
- for (Slot slot : outputSet) {
- if (!(slot instanceof SlotReference)) {
- continue;
- }
- SlotReference slotRef = (SlotReference) slot;
- if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
- uniqSlots.add(slot);
- }
- }
- builder.addUniqueSlot(uniqSlots.build());
- }
-
Review Comment:
Add a condition to check if it is a mv rather than remove those codes. They are useful if it's a trivial olapscan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on code in PR #35898:
URL: https://github.com/apache/doris/pull/35898#discussion_r1628717517
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java:
##########
@@ -455,4 +457,51 @@ public JSONObject toJson() {
olapScan.put("Properties", properties);
return olapScan;
}
+
+ @Override
+ public void computeUnique(DataTrait.Builder builder) {
+ super.computeUnique(builder);
+ if (this.selectedIndexId != getTable().getBaseIndexId()) {
+ /*
+ computing unique doesn't work for mv, because mv's key may be different from base table
+ and the key can be any expression which is difficult to deduce if it's unique. for example
+ base table:
+ CREATE TABLE IF NOT EXISTS base(
+ siteid INT(11) NOT NULL,
+ citycode SMALLINT(6) NOT NULL,
+ username VARCHAR(32) NOT NULL,
+ pv BIGINT(20) SUM NOT NULL DEFAULT '0'
+ )
+ AGGREGATE KEY (siteid,citycode,username)
+ DISTRIBUTED BY HASH(siteid) BUCKETS 5 properties("replication_num" = "1");
+
+ case1:
+ create mv1:
+ create materialized view mv1 as select siteid, sum(pv) from base group by siteid;
+ the base table siteid + citycode + username is unique but the mv1's agg key siteid is not unique
+
+ case2:
+ create mv2:
+ create materialized view mv2 as select citycode * citycode, siteid, username from base;
+ the mv2's agg key citycode * citycode is not unique
+
+ for simplicity, we disable unique compute for mv
+ */
+ return;
+ }
+ Set<Slot> outputSet = Utils.fastToImmutableSet(getOutputSet());
+ if (getTable().getKeysType().isAggregationFamily()) {
+ ImmutableSet.Builder<Slot> uniqSlots = ImmutableSet.builderWithExpectedSize(outputSet.size());
+ for (Slot slot : outputSet) {
+ if (!(slot instanceof SlotReference)) {
+ continue;
+ }
+ SlotReference slotRef = (SlotReference) slot;
+ if (slotRef.getColumn().isPresent() && slotRef.getColumn().get().isKey()) {
+ uniqSlots.add(slot);
+ }
+ }
+ builder.addUniqueSlot(uniqSlots.build());
+ }
+ }
Review Comment:
You should add the constraint code here, too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2172805230
<details>
<summary>TPC-H: <b>Total hot run time: 40528 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94f4de5c91b98645a878ff189333f169da09df94, data reload: false
------ Round 1 ----------------------------------
q1 18473 4591 4384 4384
q2 2627 193 205 193
q3 10972 1120 1133 1120
q4 10629 732 785 732
q5 7558 2715 2616 2616
q6 231 143 139 139
q7 973 623 632 623
q8 9623 2090 2059 2059
q9 8918 6457 6433 6433
q10 8941 3758 3698 3698
q11 460 241 246 241
q12 467 234 237 234
q13 18795 2951 3014 2951
q14 268 224 227 224
q15 528 486 484 484
q16 523 382 377 377
q17 965 740 645 645
q18 8011 7509 7464 7464
q19 3242 1544 1471 1471
q20 650 315 326 315
q21 4958 3782 3897 3782
q22 399 343 349 343
Total cold run time: 118211 ms
Total hot run time: 40528 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4446 4275 4228 4228
q2 367 267 256 256
q3 2990 2765 2721 2721
q4 1854 1627 1660 1627
q5 5237 5253 5240 5240
q6 216 128 128 128
q7 2158 1729 1665 1665
q8 3186 3345 3297 3297
q9 8373 8349 8343 8343
q10 3881 3691 3624 3624
q11 580 484 480 480
q12 754 575 615 575
q13 16420 2986 2985 2985
q14 303 250 276 250
q15 512 463 462 462
q16 478 431 427 427
q17 1757 1469 1462 1462
q18 7595 7504 7433 7433
q19 1973 1577 1596 1577
q20 1985 1796 1763 1763
q21 4839 4747 4735 4735
q22 620 528 562 528
Total cold run time: 70524 ms
Total hot run time: 53806 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](nereids)disable unique key compute for mv [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35898:
URL: https://github.com/apache/doris/pull/35898#issuecomment-2175669109
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org