You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HappenLee (via GitHub)" <gi...@apache.org> on 2024/01/16 07:20:43 UTC
[PR] [Performance](Join) Support all match one logic [doris]
HappenLee opened a new pull request, #30019:
URL: https://github.com/apache/doris/pull/30019
## Proposed changes
tpcds q54:
before: 3.8s
after:3.2s
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895592728
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 79aa50bfd1481ae4ea2e6977ec63bbda71a99d0a with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.8 seconds inserted 10000000 Rows, about 675K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895587871
<details>
<summary>ClickBench: <b>Total hot run time: 30.87 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79aa50bfd1481ae4ea2e6977ec63bbda71a99d0a, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.03 0.02
query3 0.22 0.05 0.05
query4 1.71 0.08 0.08
query5 0.53 0.52 0.52
query6 1.30 0.64 0.63
query7 0.02 0.01 0.01
query8 0.04 0.03 0.02
query9 0.55 0.50 0.50
query10 0.54 0.55 0.54
query11 0.12 0.10 0.08
query12 0.11 0.09 0.09
query13 0.59 0.61 0.59
query14 0.77 0.78 0.79
query15 0.78 0.77 0.76
query16 0.36 0.37 0.37
query17 1.01 0.99 1.04
query18 0.24 0.25 0.24
query19 1.89 1.77 1.79
query20 0.01 0.02 0.02
query21 15.40 0.59 0.55
query22 2.79 2.44 2.59
query23 17.56 0.77 0.79
query24 2.43 0.43 1.32
query25 0.47 0.20 0.12
query26 0.48 0.13 0.15
query27 0.05 0.05 0.05
query28 11.61 0.77 0.78
query29 12.49 3.25 3.22
query30 0.55 0.49 0.48
query31 2.78 0.34 0.35
query32 3.38 0.49 0.48
query33 3.26 3.25 3.25
query34 15.85 4.22 4.21
query35 4.28 4.33 4.24
query36 1.12 1.07 1.07
query37 0.06 0.05 0.06
query38 0.03 0.02 0.02
query39 0.02 0.01 0.01
query40 0.16 0.13 0.12
query41 0.08 0.01 0.01
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 105.78 s
Total hot run time: 30.87 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895543692
TeamCity be ut coverage result:
Function Coverage: 36.66% (8647/23589)
Line Coverage: 28.72% (70612/245845)
Region Coverage: 27.62% (36440/131951)
Branch Coverage: 24.34% (18643/76602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/79aa50bfd1481ae4ea2e6977ec63bbda71a99d0a_79aa50bfd1481ae4ea2e6977ec63bbda71a99d0a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895691581
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1893348104
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30019:
URL: https://github.com/apache/doris/pull/30019#discussion_r1455049077
##########
be/src/vec/exec/join/process_hash_table_probe_impl.h:
##########
@@ -195,9 +194,21 @@ Status ProcessHashTableProbe<JoinOpType, Parent>::do_process(HashTableType& hash
if constexpr (with_other_conjuncts || (JoinOpType != TJoinOp::RIGHT_SEMI_JOIN &&
JoinOpType != TJoinOp::RIGHT_ANTI_JOIN)) {
+ auto check_all_match_one = [](const std::vector<uint32_t>& vecs, uint32_t probe_idx,
+ int size) {
+ if (size < 1 || vecs[0] != probe_idx) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (size < 1 || vecs[0] != probe_idx) { return false;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1893397689
TeamCity be ut coverage result:
Function Coverage: 36.68% (8648/23580)
Line Coverage: 28.73% (70612/245773)
Region Coverage: 27.62% (36436/131929)
Branch Coverage: 24.34% (18640/76592)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c9fc779605a8f0fdf09be3c366e1d74915509bac_c9fc779605a8f0fdf09be3c366e1d74915509bac/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895579523
<details>
<summary>TPC-DS: <b>Total hot run time: 176270 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79aa50bfd1481ae4ea2e6977ec63bbda71a99d0a, data reload: false
query1 948 327 324 324
query2 6584 1831 1896 1831
query3 6701 207 196 196
query4 32718 22116 22185 22116
query5 6891 604 511 511
query6 251 166 174 166
query7 4609 265 259 259
query8 218 178 192 178
query9 8417 2570 2556 2556
query10 427 244 223 223
query11 16884 15535 15468 15468
query12 122 67 69 67
query13 1701 387 381 381
query14 10410 6879 6879 6879
query15 204 172 187 172
query16 4604 229 227 227
query17 1004 467 475 467
query18 1779 282 248 248
query19 175 138 133 133
query20 71 78 71 71
query21 181 131 133 131
query22 4980 4722 4619 4619
query23 31461 30764 30743 30743
query24 11491 2809 2803 2803
query25 548 316 300 300
query26 1021 142 138 138
query27 3133 273 272 272
query28 6514 1844 1838 1838
query29 1054 639 615 615
query30 289 136 139 136
query31 947 744 784 744
query32 84 51 47 47
query33 691 214 205 205
query34 1145 455 449 449
query35 888 750 763 750
query36 1327 1140 1103 1103
query37 99 56 60 56
query38 3338 3287 3221 3221
query39 1301 1276 1264 1264
query40 200 91 79 79
query41 40 36 35 35
query42 94 79 84 79
query43 508 480 506 480
query44 1099 685 704 685
query45 196 175 173 173
query46 1059 651 642 642
query47 1622 1535 1545 1535
query48 392 320 318 318
query49 1122 279 290 279
query50 671 301 308 301
query51 5367 5188 5151 5151
query52 92 77 79 77
query53 324 253 255 253
query54 889 439 438 438
query55 81 76 74 74
query56 173 169 177 169
query57 965 885 890 885
query58 188 161 159 159
query59 2865 2771 2732 2732
query60 208 193 190 190
query61 85 80 83 80
query62 627 399 353 353
query63 285 250 252 250
query64 4666 1769 1724 1724
query65 3326 3256 3250 3250
query66 1237 320 311 311
query67 15679 14933 15023 14933
query68 12217 542 545 542
query69 609 302 296 296
query70 1638 1501 1485 1485
query71 498 208 213 208
query72 5091 2829 2790 2790
query73 3005 319 314 314
query74 6949 6435 6412 6412
query75 4980 2332 2290 2290
query76 5936 1057 1084 1057
query77 661 235 240 235
query78 9668 8758 8562 8562
query79 1014 486 500 486
query80 552 324 334 324
query81 468 210 205 205
query82 200 83 79 79
query83 136 118 116 116
query84 272 67 67 67
query85 1051 321 312 312
query86 374 416 367 367
query87 3477 3376 3376 3376
query88 2758 2184 2166 2166
query89 422 333 335 333
query90 1950 187 180 180
query91 160 128 125 125
query92 51 44 43 43
query93 1546 436 446 436
query94 1164 162 157 157
query95 511 439 458 439
query96 613 323 313 313
query97 4316 4122 4170 4122
query98 208 192 187 187
query99 1022 704 713 704
Total cold run time: 292787 ms
Total hot run time: 176270 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30019:
URL: https://github.com/apache/doris/pull/30019#discussion_r1453019776
##########
be/src/vec/exec/join/process_hash_table_probe_impl.h:
##########
@@ -195,9 +194,21 @@ Status ProcessHashTableProbe<JoinOpType, Parent>::do_process(HashTableType& hash
if constexpr (with_other_conjuncts || (JoinOpType != TJoinOp::RIGHT_SEMI_JOIN &&
JoinOpType != TJoinOp::RIGHT_ANTI_JOIN)) {
+ auto check_all_match_one = [](const std::vector<uint32_t>& vecs, uint32_t probe_idx,
+ int size) {
+ if (vecs[0] != probe_idx) return false;
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (vecs[0] != probe_idx) { return false;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895393155
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895430162
TeamCity be ut coverage result:
Function Coverage: 36.65% (8646/23589)
Line Coverage: 28.72% (70595/245847)
Region Coverage: 27.60% (36425/131951)
Branch Coverage: 24.33% (18634/76602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a813a19855ecaccd5060cb0e9706bd65fd753247_a813a19855ecaccd5060cb0e9706bd65fd753247/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #30019:
URL: https://github.com/apache/doris/pull/30019#discussion_r1455142319
##########
be/src/vec/exec/join/process_hash_table_probe_impl.h:
##########
@@ -188,16 +185,27 @@ Status ProcessHashTableProbe<JoinOpType, Parent>::do_process(HashTableType& hash
probe_index = new_probe_idx;
build_index = new_build_idx;
current_offset = new_current_offset;
- probe_size = probe_index - last_probe_index;
}
build_side_output_column(mcol, *_right_output_slot_flags, current_offset, with_other_conjuncts);
if constexpr (with_other_conjuncts || (JoinOpType != TJoinOp::RIGHT_SEMI_JOIN &&
JoinOpType != TJoinOp::RIGHT_ANTI_JOIN)) {
+ auto check_all_match_one = [](const std::vector<uint32_t>& vecs, uint32_t probe_idx,
+ int size) {
+ if (size < 1 || vecs[0] != probe_idx) return false;
+ for (int i = 1; i < size; i++) {
Review Comment:
warning: statement should be inside braces [readability-braces-around-statements]
```suggestion
if (size < 1 || vecs[0] != probe_idx) { return false;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895560966
<details>
<summary>TPC-H: <b>Total hot run time: 38704 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79aa50bfd1481ae4ea2e6977ec63bbda71a99d0a, data reload: false
------ Round 1 ----------------------------------
q1 17888 5089 5284 5089
q2 2038 140 131 131
q3 10826 1164 1141 1141
q4 10446 735 846 735
q5 7761 3207 3117 3117
q6 202 122 121 121
q7 868 493 478 478
q8 9248 1938 1950 1938
q9 7271 6411 6305 6305
q10 8245 3079 3073 3073
q11 412 209 205 205
q12 358 195 198 195
q13 17987 3363 3340 3340
q14 249 207 205 205
q15 554 510 508 508
q16 428 378 369 369
q17 964 541 523 523
q18 7589 7075 6863 6863
q19 3102 1363 1414 1363
q20 607 276 299 276
q21 2768 2428 2422 2422
q22 355 311 307 307
Total cold run time: 110166 ms
Total hot run time: 38704 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5635 5247 5252 5247
q2 328 217 212 212
q3 3353 3229 3269 3229
q4 2107 2033 2063 2033
q5 6021 5851 5825 5825
q6 203 116 114 114
q7 2294 1840 1825 1825
q8 3241 3375 3406 3375
q9 8970 8849 8853 8849
q10 3933 3819 3890 3819
q11 554 446 438 438
q12 816 631 618 618
q13 16961 3134 3132 3132
q14 289 256 261 256
q15 551 507 514 507
q16 522 451 469 451
q17 1882 1798 1814 1798
q18 9566 16102 9786 9786
q19 20349 1554 1488 1488
q20 4586 1934 1931 1931
q21 14422 5380 5383 5380
q22 980 533 556 533
Total cold run time: 107563 ms
Total hot run time: 60846 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895474993
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30019:
URL: https://github.com/apache/doris/pull/30019#issuecomment-1895686906
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Performance](Join) Support all match one logic [doris]
Posted by "BiteTheDDDDt (via GitHub)" <gi...@apache.org>.
BiteTheDDDDt merged PR #30019:
URL: https://github.com/apache/doris/pull/30019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org