You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "XieJiann (via GitHub)" <gi...@apache.org> on 2024/01/23 05:26:17 UTC
[PR] [feat](Nereids): refactor elimanate inner join by pkfk [doris]
XieJiann opened a new pull request, #30258:
URL: https://github.com/apache/doris/pull/30258
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909277147
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a21cbfaf8dbecbb60d173af10e9a37d4e26fbbea with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.8 seconds inserted 10000000 Rows, about 675K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): refactor elimanate inner join by pkfk [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1907588569
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "seawinde (via GitHub)" <gi...@apache.org>.
seawinde commented on code in PR #30258:
URL: https://github.com/apache/doris/pull/30258#discussion_r1465779427
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/util/JoinUtils.java:
##########
@@ -279,11 +282,46 @@ private static List<Slot> applyNullable(List<Slot> slots, boolean nullable) {
.collect(ImmutableList.toImmutableList());
}
+ private static Map<Slot, Slot> mapPrimaryToForeign(ImmutableEqualSet<Slot> equivalenceSet,
+ Set<Slot> foreignKeys) {
+ ImmutableMap.Builder<Slot, Slot> builder = new ImmutableMap.Builder<>();
+ for (Slot foreignSlot : foreignKeys) {
+ Set<Slot> primarySlots = equivalenceSet.calEqualSet(foreignSlot);
+ if (primarySlots.size() != 1) {
+ return ImmutableMap.of();
+ }
+ builder.put(primarySlots.iterator().next(), foreignSlot);
+ }
+ return builder.build();
+ }
+
+ /**
+ * Check whether the given join can be eliminated by pk-fk
+ */
+ public static boolean canEliminateByFk(LogicalJoin<?, ?> join, Plan primaryPlan, Plan foreignPlan) {
+ if (!join.getJoinType().isInnerJoin() || !join.getOtherJoinConjuncts().isEmpty() || join.isMarkJoin()) {
+ return false;
+ }
+
+ ForeignKeyContext context = new ForeignKeyContext();
+ context.collectForeignKeyConstraint(primaryPlan);
+ context.collectForeignKeyConstraint(foreignPlan);
+
+ ImmutableEqualSet<Slot> equalSet = join.getEqualSlots();
+ Set<Slot> primaryKey = Sets.intersection(equalSet.getAllItemSet(), primaryPlan.getOutputSet());
+ Set<Slot> foreignKey = Sets.intersection(equalSet.getAllItemSet(), foreignPlan.getOutputSet());
+ if (!context.isForeignKey(foreignKey) && !context.isPrimaryKey(primaryKey)) {
Review Comment:
maybe this should be ||（or）?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "jackwener (via GitHub)" <gi...@apache.org>.
jackwener merged PR #30258:
URL: https://github.com/apache/doris/pull/30258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): refactor elimanate inner join by pkfk [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1907587197
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909323431
<details>
<summary>TPC-H: <b>Total hot run time: 38561 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9af1452eba8453fa2503cea8814a2fffcd9cd19f, data reload: false
------ Round 1 ----------------------------------
q1 17704 5087 5253 5087
q2 2061 136 128 128
q3 10649 1153 1177 1153
q4 10225 754 833 754
q5 7721 3115 3104 3104
q6 195 118 120 118
q7 853 504 474 474
q8 9256 1961 1942 1942
q9 7208 6381 6322 6322
q10 8260 3058 3090 3058
q11 422 210 204 204
q12 356 183 187 183
q13 17990 3367 3364 3364
q14 245 215 217 215
q15 558 533 492 492
q16 469 378 364 364
q17 940 549 493 493
q18 7440 6879 6819 6819
q19 2909 1336 1439 1336
q20 593 279 304 279
q21 2761 2366 2396 2366
q22 341 306 312 306
Total cold run time: 109156 ms
Total hot run time: 38561 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5366 5361 5962 5361
q2 319 214 212 212
q3 3339 3238 3230 3230
q4 2046 2056 2037 2037
q5 6059 5924 5897 5897
q6 194 115 118 115
q7 2302 1850 1803 1803
q8 3235 3363 3395 3363
q9 8916 8899 8783 8783
q10 3881 3812 3830 3812
q11 558 465 452 452
q12 816 608 637 608
q13 16913 3190 3147 3147
q14 281 259 275 259
q15 547 495 492 492
q16 520 458 472 458
q17 1866 1819 1842 1819
q18 9500 17814 9614 9614
q19 25628 1605 1547 1547
q20 4688 1937 1924 1924
q21 14496 5351 5328 5328
q22 1003 528 513 513
Total cold run time: 112473 ms
Total hot run time: 60774 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909295151
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909332915
<details>
<summary>TPC-DS: <b>Total hot run time: 186133 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9af1452eba8453fa2503cea8814a2fffcd9cd19f, data reload: false
query1 938 331 330 330
query2 6594 1870 1928 1870
query3 6712 199 202 199
query4 31442 22131 22160 22131
query5 4448 446 362 362
query6 244 157 149 149
query7 4609 263 263 263
query8 222 175 197 175
query9 8365 2628 2602 2602
query10 437 249 245 245
query11 16908 15502 15327 15327
query12 124 71 64 64
query13 1683 372 379 372
query14 10679 6954 6976 6954
query15 213 179 181 179
query16 5791 258 261 258
query17 942 467 476 467
query18 1777 252 251 251
query19 174 128 130 128
query20 69 74 68 68
query21 189 127 135 127
query22 5025 4883 4696 4696
query23 31500 30873 31011 30873
query24 11703 2785 2858 2785
query25 545 310 307 307
query26 942 147 144 144
query27 2912 290 286 286
query28 6333 1838 1831 1831
query29 1010 629 614 614
query30 279 138 145 138
query31 939 751 764 751
query32 79 50 47 47
query33 695 215 211 211
query34 1136 456 480 456
query35 915 776 744 744
query36 1295 1128 1157 1128
query37 89 59 57 57
query38 3354 3221 3256 3221
query39 1321 1276 1278 1276
query40 195 90 85 85
query41 36 35 34 34
query42 85 81 91 81
query43 508 481 484 481
query44 1088 692 708 692
query45 200 178 175 175
query46 1085 667 649 649
query47 1686 1549 1582 1549
query48 394 322 305 305
query49 1128 291 283 283
query50 699 323 309 309
query51 5324 5190 5207 5190
query52 90 76 75 75
query53 318 265 274 265
query54 246 181 185 181
query55 81 77 80 77
query56 172 158 159 158
query57 1031 950 907 907
query58 177 151 159 151
query59 2900 2550 2706 2550
query60 200 178 179 178
query61 80 80 83 80
query62 632 359 350 350
query63 300 258 260 258
query64 4620 1830 1758 1758
query65 3364 3282 3277 3277
query66 1180 324 311 311
query67 15589 15365 14992 14992
query68 12975 549 523 523
query69 611 293 302 293
query70 1846 1536 1498 1498
query71 10423 10195 10193 10193
query72 5133 2835 2838 2835
query73 3305 319 318 318
query74 6887 6340 6452 6340
query75 4944 2337 2295 2295
query76 6291 1073 1026 1026
query77 678 227 221 221
query78 9059 8869 8532 8532
query79 2632 498 492 492
query80 571 318 328 318
query81 473 205 205 205
query82 209 79 79 79
query83 135 113 114 113
query84 265 67 67 67
query85 1087 325 325 325
query86 414 374 387 374
query87 3524 3337 3381 3337
query88 3160 2212 2182 2182
query89 453 366 363 363
query90 1972 186 183 183
query91 152 120 122 120
query92 53 47 43 43
query93 1862 424 419 419
query94 1306 158 154 154
query95 503 457 448 448
query96 619 317 321 317
query97 4256 4138 4150 4138
query98 198 191 195 191
query99 981 715 704 704
Total cold run time: 303017 ms
Total hot run time: 186133 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1907646885
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit b1c3778a04e64ed57729ca9244b5425c260bf526 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.8 seconds inserted 10000000 Rows, about 675K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909339862
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 9af1452eba8453fa2503cea8814a2fffcd9cd19f with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 14.9 seconds inserted 10000000 Rows, about 671K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1907633662
<details>
<summary>TPC-DS: <b>Total hot run time: 186361 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1c3778a04e64ed57729ca9244b5425c260bf526, data reload: false
query1 920 337 328 328
query2 6559 2163 1903 1903
query3 6701 200 194 194
query4 31282 22371 22039 22039
query5 4450 367 438 367
query6 231 154 159 154
query7 4600 277 257 257
query8 229 173 178 173
query9 8377 2512 2482 2482
query10 412 214 227 214
query11 16746 15611 15547 15547
query12 124 66 68 66
query13 1681 375 385 375
query14 10606 6930 6869 6869
query15 213 179 177 177
query16 5795 259 251 251
query17 943 481 474 474
query18 1777 251 247 247
query19 173 131 128 128
query20 74 75 69 69
query21 190 134 124 124
query22 4942 4846 4705 4705
query23 31686 30838 30738 30738
query24 12921 2804 2807 2804
query25 575 317 319 317
query26 1833 145 147 145
query27 3591 289 288 288
query28 7658 1828 1822 1822
query29 2260 620 629 620
query30 283 139 137 137
query31 957 752 770 752
query32 79 49 51 49
query33 711 215 216 215
query34 1262 455 458 455
query35 909 721 774 721
query36 1424 1258 1173 1173
query37 175 58 59 58
query38 3372 3267 3264 3264
query39 1314 1281 1269 1269
query40 349 87 84 84
query41 39 35 36 35
query42 92 81 87 81
query43 503 470 463 463
query44 1103 687 682 682
query45 192 177 171 171
query46 1072 651 656 651
query47 1649 1581 1558 1558
query48 402 316 317 316
query49 1216 289 291 289
query50 698 318 311 311
query51 5317 5198 5259 5198
query52 98 77 82 77
query53 328 247 272 247
query54 245 193 184 184
query55 84 77 74 74
query56 176 162 169 162
query57 1005 899 915 899
query58 187 164 157 157
query59 2836 2745 2749 2745
query60 207 187 196 187
query61 83 82 83 82
query62 601 383 366 366
query63 294 275 262 262
query64 6036 1770 1737 1737
query65 3315 3254 3247 3247
query66 1442 323 313 313
query67 15667 15041 15485 15041
query68 11239 540 547 540
query69 616 300 304 300
query70 1609 1552 1550 1550
query71 10494 10199 10194 10194
query72 4889 2816 2822 2816
query73 1983 315 318 315
query74 6961 6442 6343 6343
query75 5086 2351 2261 2261
query76 7212 981 1008 981
query77 887 240 236 236
query78 10022 8895 8642 8642
query79 1008 499 500 499
query80 640 320 308 308
query81 447 208 199 199
query82 207 80 76 76
query83 158 116 117 116
query84 266 76 70 70
query85 1082 346 309 309
query86 393 372 372 372
query87 3488 3351 3369 3351
query88 2943 2197 2198 2197
query89 430 365 353 353
query90 2143 185 183 183
query91 153 132 126 126
query92 56 42 44 42
query93 1345 450 431 431
query94 1307 160 158 158
query95 509 460 464 460
query96 613 324 317 317
query97 4243 4111 4164 4111
query98 201 193 190 190
query99 1005 736 726 726
Total cold run time: 306906 ms
Total hot run time: 186361 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): refactor elimanate inner join by pkfk [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1905686008
> add some regression case, please
this has not been finished, I'll add some test in it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909227470
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909275053
<details>
<summary>ClickBench: <b>Total hot run time: 30.98 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a21cbfaf8dbecbb60d173af10e9a37d4e26fbbea, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.03 0.02
query3 0.23 0.05 0.04
query4 1.70 0.08 0.08
query5 0.53 0.53 0.53
query6 1.28 0.65 0.63
query7 0.02 0.02 0.01
query8 0.04 0.03 0.02
query9 0.55 0.50 0.50
query10 0.56 0.57 0.58
query11 0.11 0.08 0.08
query12 0.11 0.10 0.09
query13 0.61 0.62 0.61
query14 0.78 0.81 0.80
query15 0.80 0.78 0.77
query16 0.40 0.37 0.38
query17 1.01 1.00 1.01
query18 0.22 0.25 0.24
query19 1.87 1.80 1.79
query20 0.01 0.01 0.01
query21 15.40 0.57 0.56
query22 2.14 2.39 1.79
query23 17.18 0.88 0.82
query24 2.80 1.09 0.96
query25 0.42 0.15 0.15
query26 0.58 0.12 0.13
query27 0.06 0.06 0.05
query28 11.44 0.78 0.77
query29 12.52 3.13 3.16
query30 0.57 0.49 0.47
query31 2.78 0.35 0.35
query32 3.37 0.49 0.48
query33 3.21 3.27 3.19
query34 15.74 4.40 4.33
query35 4.39 4.42 4.27
query36 1.12 1.09 1.08
query37 0.06 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.02
query40 0.16 0.13 0.13
query41 0.07 0.02 0.01
query42 0.02 0.02 0.02
query43 0.03 0.01 0.02
Total cold run time: 105.04 s
Total hot run time: 30.98 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909295327
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): refactor elimanate inner join by pkfk [doris]
Posted by "XieJiann (via GitHub)" <gi...@apache.org>.
XieJiann commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1905687062
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909262969
<details>
<summary>TPC-H: <b>Total hot run time: 39153 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a21cbfaf8dbecbb60d173af10e9a37d4e26fbbea, data reload: false
------ Round 1 ----------------------------------
q1 17704 5358 5345 5345
q2 2050 143 136 136
q3 10635 1159 1156 1156
q4 10252 805 775 775
q5 7744 3103 3136 3103
q6 201 124 123 123
q7 856 504 488 488
q8 9287 1938 1943 1938
q9 7204 6414 6370 6370
q10 8259 3055 3031 3031
q11 422 206 206 206
q12 356 188 189 188
q13 18012 3366 3368 3366
q14 248 222 207 207
q15 548 515 497 497
q16 467 372 365 365
q17 948 546 534 534
q18 7568 6888 6942 6888
q19 2285 1364 1353 1353
q20 560 310 301 301
q21 2762 2477 2484 2477
q22 352 311 306 306
Total cold run time: 108720 ms
Total hot run time: 39153 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5475 5294 5195 5195
q2 330 212 215 212
q3 3369 3256 3249 3249
q4 2094 2071 2059 2059
q5 6124 5859 5858 5858
q6 200 113 123 113
q7 2278 1887 1867 1867
q8 3225 3373 3416 3373
q9 9002 8872 8923 8872
q10 3951 3850 3815 3815
q11 559 457 462 457
q12 802 639 617 617
q13 16901 3168 3143 3143
q14 283 259 257 257
q15 552 504 508 504
q16 535 447 475 447
q17 1894 1838 1914 1838
q18 9368 9673 10863 9673
q19 21228 1566 1509 1509
q20 4371 1943 1927 1927
q21 13739 5520 5375 5375
q22 995 521 555 521
Total cold run time: 107275 ms
Total hot run time: 60881 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909337142
<details>
<summary>ClickBench: <b>Total hot run time: 30.12 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9af1452eba8453fa2503cea8814a2fffcd9cd19f, data reload: false
query1 0.04 0.03 0.02
query2 0.06 0.02 0.02
query3 0.23 0.05 0.05
query4 1.70 0.08 0.07
query5 0.53 0.52 0.53
query6 1.34 0.64 0.64
query7 0.02 0.02 0.01
query8 0.04 0.03 0.02
query9 0.54 0.50 0.50
query10 0.56 0.57 0.56
query11 0.12 0.08 0.08
query12 0.10 0.09 0.09
query13 0.60 0.62 0.60
query14 0.77 0.81 0.81
query15 0.78 0.78 0.77
query16 0.40 0.39 0.37
query17 1.05 1.00 1.01
query18 0.23 0.25 0.24
query19 1.87 1.79 1.81
query20 0.01 0.01 0.01
query21 15.63 0.57 0.56
query22 2.82 2.24 1.48
query23 17.44 0.73 0.77
query24 2.37 0.89 0.65
query25 0.37 0.24 0.15
query26 0.48 0.14 0.13
query27 0.04 0.05 0.06
query28 12.23 0.76 0.77
query29 12.50 3.08 3.12
query30 0.57 0.50 0.51
query31 2.78 0.35 0.35
query32 3.36 0.47 0.49
query33 3.20 3.23 3.18
query34 15.80 4.22 4.36
query35 4.36 4.31 4.31
query36 1.11 1.07 1.07
query37 0.07 0.05 0.05
query38 0.04 0.03 0.03
query39 0.02 0.02 0.01
query40 0.16 0.13 0.13
query41 0.08 0.01 0.02
query42 0.03 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 106.47 s
Total hot run time: 30.12 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909407862
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1909271419
<details>
<summary>TPC-DS: <b>Total hot run time: 187097 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a21cbfaf8dbecbb60d173af10e9a37d4e26fbbea, data reload: false
query1 944 332 330 330
query2 6546 1883 1794 1794
query3 6704 208 195 195
query4 32528 22104 22070 22070
query5 4453 458 415 415
query6 241 158 160 158
query7 4621 262 255 255
query8 224 176 177 176
query9 8481 2653 2625 2625
query10 415 249 226 226
query11 17335 15644 15688 15644
query12 120 70 68 68
query13 1678 386 381 381
query14 10736 7177 7091 7091
query15 222 179 188 179
query16 5787 262 256 256
query17 937 479 472 472
query18 1783 265 254 254
query19 177 138 128 128
query20 71 77 68 68
query21 187 136 131 131
query22 5041 4777 4627 4627
query23 32378 31780 31534 31534
query24 12625 2805 2841 2805
query25 584 313 306 306
query26 1833 146 145 145
query27 3163 288 287 287
query28 7418 1854 1835 1835
query29 2207 627 632 627
query30 279 136 136 136
query31 934 725 747 725
query32 82 51 52 51
query33 711 216 211 211
query34 1148 459 479 459
query35 901 816 797 797
query36 1278 1192 1202 1192
query37 94 61 62 61
query38 3334 3187 3210 3187
query39 1323 1268 1263 1263
query40 344 85 82 82
query41 37 35 35 35
query42 83 79 79 79
query43 515 484 465 465
query44 1079 699 699 699
query45 191 177 174 174
query46 1070 687 651 651
query47 1700 1583 1451 1451
query48 376 301 303 301
query49 1226 293 293 293
query50 693 320 322 320
query51 5305 5229 5176 5176
query52 88 80 80 80
query53 327 271 261 261
query54 240 174 191 174
query55 80 73 78 73
query56 173 164 167 164
query57 1017 939 902 902
query58 185 151 155 151
query59 2909 2722 2638 2638
query60 212 178 182 178
query61 84 94 85 85
query62 611 366 359 359
query63 281 257 271 257
query64 6071 1777 1759 1759
query65 3353 3286 3258 3258
query66 1393 319 316 316
query67 15538 14866 15261 14866
query68 12550 519 517 517
query69 602 292 295 292
query70 1710 1481 1441 1441
query71 10479 10201 10200 10200
query72 4883 2851 2865 2851
query73 2641 322 318 318
query74 7077 6299 6386 6299
query75 5052 2363 2371 2363
query76 6675 1004 1046 1004
query77 710 224 242 224
query78 9092 8645 8584 8584
query79 1008 488 517 488
query80 590 332 314 314
query81 452 205 202 202
query82 204 83 80 80
query83 144 125 117 117
query84 272 69 68 68
query85 1118 324 339 324
query86 392 384 400 384
query87 3564 3326 3310 3310
query88 3158 2206 2219 2206
query89 430 357 354 354
query90 1991 191 187 187
query91 153 129 134 129
query92 59 44 42 42
query93 1357 461 413 413
query94 1260 166 158 158
query95 499 470 457 457
query96 613 323 320 320
query97 4276 4175 4216 4175
query98 196 206 186 186
query99 1030 686 679 679
Total cold run time: 308941 ms
Total hot run time: 187097 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1907616162
<details>
<summary>TPC-H: <b>Total hot run time: 39015 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1c3778a04e64ed57729ca9244b5425c260bf526, data reload: false
------ Round 1 ----------------------------------
q1 17962 5437 5487 5437
q2 2282 163 138 138
q3 11040 1198 1167 1167
q4 10621 803 781 781
q5 8221 3223 3189 3189
q6 203 123 123 123
q7 893 524 474 474
q8 9214 1950 1949 1949
q9 7297 6357 6391 6357
q10 8201 3034 3019 3019
q11 412 209 206 206
q12 357 193 191 191
q13 17980 3405 3329 3329
q14 253 219 218 218
q15 560 504 501 501
q16 420 377 379 377
q17 928 494 498 494
q18 7584 6746 6907 6746
q19 1568 1357 1398 1357
q20 577 299 314 299
q21 2774 2428 2365 2365
q22 360 308 298 298
Total cold run time: 109707 ms
Total hot run time: 39015 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5335 5153 5237 5153
q2 322 216 217 216
q3 3339 3256 3271 3256
q4 2120 2038 2031 2031
q5 5906 5822 5995 5822
q6 199 114 117 114
q7 2314 1814 1945 1814
q8 3215 3396 3377 3377
q9 8938 8866 8824 8824
q10 3967 3846 3833 3833
q11 566 462 436 436
q12 801 661 610 610
q13 16905 3157 3163 3157
q14 287 251 273 251
q15 553 509 508 508
q16 520 469 496 469
q17 1880 1878 1842 1842
q18 9606 18955 9714 9714
q19 23751 1555 1516 1516
q20 4552 1965 1918 1918
q21 14663 5413 5364 5364
q22 994 582 576 576
Total cold run time: 110733 ms
Total hot run time: 60801 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): eliminate inner join by pk fk when comparing mv [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1907641284
<details>
<summary>ClickBench: <b>Total hot run time: 30.53 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b1c3778a04e64ed57729ca9244b5425c260bf526, data reload: false
query1 0.03 0.03 0.02
query2 0.06 0.03 0.02
query3 0.23 0.05 0.05
query4 1.69 0.06 0.06
query5 0.54 0.52 0.53
query6 1.27 0.63 0.62
query7 0.02 0.02 0.01
query8 0.04 0.03 0.03
query9 0.55 0.49 0.49
query10 0.58 0.55 0.55
query11 0.12 0.09 0.08
query12 0.11 0.09 0.09
query13 0.61 0.60 0.60
query14 0.79 0.80 0.81
query15 0.79 0.77 0.77
query16 0.37 0.39 0.39
query17 1.04 1.01 1.03
query18 0.23 0.26 0.23
query19 1.84 1.78 1.78
query20 0.02 0.01 0.01
query21 15.41 0.58 0.59
query22 2.73 2.73 1.32
query23 17.12 0.81 0.78
query24 2.44 1.19 1.34
query25 0.46 0.21 0.15
query26 0.56 0.14 0.15
query27 0.06 0.05 0.04
query28 11.09 0.79 0.75
query29 12.55 3.08 3.16
query30 0.57 0.47 0.49
query31 2.79 0.36 0.34
query32 3.40 0.48 0.48
query33 3.22 3.23 3.23
query34 15.70 4.22 4.27
query35 4.39 4.32 4.29
query36 1.10 1.09 1.08
query37 0.07 0.05 0.05
query38 0.03 0.03 0.03
query39 0.02 0.02 0.02
query40 0.15 0.15 0.13
query41 0.07 0.02 0.01
query42 0.03 0.01 0.02
query43 0.02 0.02 0.01
Total cold run time: 104.91 s
Total hot run time: 30.53 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feat](Nereids): refactor elimanate inner join by pkfk [doris]
Posted by "englefly (via GitHub)" <gi...@apache.org>.
englefly commented on PR #30258:
URL: https://github.com/apache/doris/pull/30258#issuecomment-1905433984
add some regression case, please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org