You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HappenLee (via GitHub)" <gi...@apache.org> on 2024/01/02 18:23:32 UTC
[PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
HappenLee opened a new pull request, #29439:
URL: https://github.com/apache/doris/pull/29439
## Proposed changes
Beforeļ¼
```
RuntimeFilter: (id = 5, type = bloomfilter):
- Info: [IsPushDown = true, RuntimeFilterState = READY, IsIgnored = false, HasRemoteTarget = true, HasLocalTarget = false]
- EffectTime: 811 ms
```
After:
```
RuntimeFilter: (id = 5, type = bloomfilter):
- Info: [IsPushDown = true, RuntimeFilterState = READY, IsIgnored = false, HasRemoteTarget = true, HasLocalTarget = false]
- EffectTime: 50 ms
```
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29439:
URL: https://github.com/apache/doris/pull/29439#issuecomment-1874798447
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on PR #29439:
URL: https://github.com/apache/doris/pull/29439#issuecomment-1874386188
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29439:
URL: https://github.com/apache/doris/pull/29439#issuecomment-1874444095
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.42 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17183460324 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29439:
URL: https://github.com/apache/doris/pull/29439#issuecomment-1874431684
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 9fdcd3ee6f1470b40f83a0bc77b3ea2cae4342c6, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5486 5194 5169 5169
q2 400 195 158 158
q3 1476 1207 1309 1207
q4 1095 843 818 818
q5 3161 3127 3094 3094
q6 226 149 135 135
q7 994 575 534 534
q8 2134 2191 2238 2191
q9 6709 6699 6706 6699
q10 3200 3159 3109 3109
q11 367 221 220 220
q12 391 245 249 245
q13 4455 3677 3721 3677
q14 247 215 219 215
q15 605 552 534 534
q16 487 410 419 410
q17 1036 516 499 499
q18 7094 6783 6813 6783
q19 1655 1523 1468 1468
q20 565 352 353 352
q21 2933 2529 2465 2465
q22 408 311 326 311
Total cold run time: 45124 ms
Total hot run time: 40293 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5081 5054 5099 5054
q2 337 258 260 258
q3 3394 3332 3322 3322
q4 2160 2011 2036 2011
q5 6010 5981 5970 5970
q6 228 129 129 129
q7 2437 1978 1974 1974
q8 3575 3696 3700 3696
q9 9108 9045 9022 9022
q10 3926 3921 3927 3921
q11 621 500 478 478
q12 821 658 664 658
q13 3894 3240 3269 3240
q14 314 270 297 270
q15 613 553 541 541
q16 552 517 534 517
q17 2035 1848 1823 1823
q18 8764 8383 8779 8383
q19 1730 1674 1703 1674
q20 2336 2006 1982 1982
q21 5784 5435 5360 5360
q22 578 484 526 484
Total cold run time: 64298 ms
Total hot run time: 60767 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #29439:
URL: https://github.com/apache/doris/pull/29439#discussion_r1439710313
##########
be/src/exprs/runtime_filter_slots.h:
##########
@@ -204,10 +204,10 @@ class VRuntimeFilterSlots {
}
// publish runtime filter
- Status publish() {
+ Status publish(bool publish_local = false) {
Review Comment:
warning: method 'publish' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status publish(bool publish_local = false) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #29439:
URL: https://github.com/apache/doris/pull/29439#discussion_r1440016363
##########
be/src/exprs/runtime_filter.cpp:
##########
@@ -1010,13 +1010,17 @@ Status IRuntimeFilter::publish() {
filter->update_runtime_filter_type_to_profile();
filter->signal();
}
- return Status::OK();
- } else {
+ } else if (!publish_local) {
TNetworkAddress addr;
DCHECK(_state != nullptr);
RETURN_IF_ERROR(_state->runtime_filter_mgr->get_merge_addr(&addr));
return push_to_remote(_state, &addr, _opt_remote_rf);
+ } else {
+ // remote broadcast join only push onetime in build shared hash table
+ // publish_local only set true on copy shared hash table
+ DCHECK(_is_broadcast_join);
Review Comment:
This will cause correctness problem if multiple targets on different BE should be broadcasted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29439:
URL: https://github.com/apache/doris/pull/29439#issuecomment-1874798464
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #29439:
URL: https://github.com/apache/doris/pull/29439
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Opt](rf) Opt broadcast join remote runtime filter merge and wait [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29439:
URL: https://github.com/apache/doris/pull/29439#issuecomment-1874431815
TeamCity be ut coverage result:
Function Coverage: 36.61% (8613/23524)
Line Coverage: 28.67% (69990/244142)
Region Coverage: 27.65% (36227/131021)
Branch Coverage: 24.35% (18510/76014)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9fdcd3ee6f1470b40f83a0bc77b3ea2cae4342c6_9fdcd3ee6f1470b40f83a0bc77b3ea2cae4342c6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org