You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mymeiyi (via GitHub)" <gi...@apache.org> on 2025/04/07 03:18:15 UTC
[PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
mymeiyi opened a new pull request, #49822:
URL: https://github.com/apache/doris/pull/49822
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #49822:
URL: https://github.com/apache/doris/pull/49822#discussion_r2030961914
##########
be/src/pipeline/dependency.h:
##########
@@ -204,7 +204,7 @@ struct RuntimeFilterTimerQueue;
class RuntimeFilterTimer {
public:
RuntimeFilterTimer(int64_t registration_time, int32_t wait_time_ms,
- std::shared_ptr<RuntimeFilterDependency> parent)
+ std::shared_ptr<Dependency> parent)
Review Comment:
timedBlockingDetector
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #49822:
URL: https://github.com/apache/doris/pull/49822#discussion_r2096710660
##########
be/src/pipeline/exec/group_commit_scan_operator.cpp:
##########
@@ -46,8 +45,21 @@ Status GroupCommitLocalState::init(RuntimeState* state, LocalStateInfo& info) {
auto& p = _parent->cast<GroupCommitOperatorX>();
_get_block_dependency = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
"GroupCommitGetBlockDependency", true);
- return state->exec_env()->group_commit_mgr()->get_load_block_queue(
+ _timer_dependency = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
+ "GroupCommitTimerDependency", true);
+ auto st = state->exec_env()->group_commit_mgr()->get_load_block_queue(
p._table_id, state->fragment_instance_id(), load_block_queue, _get_block_dependency);
+ if (st.ok()) {
+ DCHECK(load_block_queue != nullptr);
+ _timer_dependency->block();
+ _runtime_filter_timer = std::make_shared<pipeline::RuntimeFilterTimer>(
+ MonotonicMillis(), load_block_queue->get_group_commit_interval_ms(),
+ _timer_dependency);
+ std::vector<std::shared_ptr<pipeline::RuntimeFilterTimer>> timers;
Review Comment:
这种地方都得用注释写清楚，否则，别人还以为我们的group commit 跟runtime filter 有啥关联
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2889727502
# BE Regression && UT Coverage Report
Increment line coverage `95.45% (21/22)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49822_18509efc38e4dc855667894f374bce30e7973650_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49822_18509efc38e4dc855667894f374bce30e7973650_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 79.39% (20845/26258) |
| Line Coverage | 72.62% (214775/295750) |
| Region Coverage | 70.82% (126361/178417) |
| Branch Coverage | 64.60% (65501/101398) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2795790347
<details>
<summary>TPC-H: <b>Total hot run time: 35526 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a023e7737c776901c1893fd5eea3c26e314f3df8, data reload: false
------ Round 1 ----------------------------------
q1 26046 5310 5275 5275
q2 2081 293 187 187
q3 10375 1394 737 737
q4 10240 1079 560 560
q5 7582 2729 2649 2649
q6 204 174 133 133
q7 1006 793 652 652
q8 9330 1425 1258 1258
q9 7078 5311 5314 5311
q10 6944 2367 1929 1929
q11 519 293 283 283
q12 382 390 227 227
q13 17796 3832 3152 3152
q14 248 236 217 217
q15 561 508 488 488
q16 666 628 597 597
q17 623 902 360 360
q18 7591 7147 7156 7147
q19 1570 1090 587 587
q20 347 366 222 222
q21 4317 3693 2568 2568
q22 1086 1058 987 987
Total cold run time: 116592 ms
Total hot run time: 35526 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5593 5520 5506 5506
q2 253 348 244 244
q3 2255 2761 2485 2485
q4 1628 2100 1557 1557
q5 4633 4468 4420 4420
q6 264 183 136 136
q7 2092 1981 1804 1804
q8 2891 2809 2777 2777
q9 7154 7105 7133 7105
q10 3164 3296 2802 2802
q11 625 517 506 506
q12 713 781 640 640
q13 3658 4153 3388 3388
q14 290 305 287 287
q15 552 485 478 478
q16 679 704 645 645
q17 1231 1782 1407 1407
q18 7762 7613 7460 7460
q19 861 865 1150 865
q20 1988 2073 1896 1896
q21 5630 5032 4982 4982
q22 1111 1083 1044 1044
Total cold run time: 55027 ms
Total hot run time: 52434 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782109591
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2889571999
<details>
<summary>ClickBench: <b>Total hot run time: 28.74 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18509efc38e4dc855667894f374bce30e7973650, data reload: false
query1 0.03 0.03 0.03
query2 0.13 0.10 0.11
query3 0.25 0.19 0.19
query4 1.60 0.19 0.10
query5 0.45 0.40 0.43
query6 1.15 0.66 0.67
query7 0.03 0.01 0.01
query8 0.04 0.03 0.03
query9 0.58 0.52 0.52
query10 0.56 0.58 0.57
query11 0.15 0.11 0.11
query12 0.14 0.11 0.11
query13 0.62 0.59 0.60
query14 0.79 0.81 0.81
query15 0.87 0.85 0.85
query16 0.39 0.38 0.40
query17 1.06 1.04 1.02
query18 0.22 0.21 0.20
query19 1.95 1.82 1.84
query20 0.02 0.01 0.01
query21 15.42 0.87 0.53
query22 0.76 1.21 0.63
query23 14.94 1.39 0.61
query24 6.65 1.06 0.51
query25 0.52 0.25 0.06
query26 0.55 0.16 0.15
query27 0.04 0.05 0.06
query28 9.24 0.91 0.44
query29 12.53 3.98 3.36
query30 0.26 0.10 0.07
query31 2.81 0.58 0.38
query32 3.23 0.55 0.46
query33 3.09 3.10 3.07
query34 15.84 5.12 4.55
query35 4.57 4.56 4.60
query36 0.67 0.50 0.48
query37 0.08 0.06 0.06
query38 0.06 0.04 0.03
query39 0.03 0.02 0.02
query40 0.17 0.14 0.14
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 102.64 s
Total hot run time: 28.74 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2889550119
<details>
<summary>TPC-H: <b>Total hot run time: 33561 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18509efc38e4dc855667894f374bce30e7973650, data reload: false
------ Round 1 ----------------------------------
q1 25928 5617 4961 4961
q2 2080 266 187 187
q3 10410 1228 684 684
q4 10220 997 522 522
q5 7694 2383 2318 2318
q6 174 159 130 130
q7 900 737 615 615
q8 9379 1356 1089 1089
q9 7106 5084 5215 5084
q10 6879 2303 1887 1887
q11 505 291 260 260
q12 337 347 221 221
q13 17754 3617 3035 3035
q14 236 224 212 212
q15 529 486 476 476
q16 424 430 372 372
q17 574 851 358 358
q18 7477 7097 7239 7097
q19 1537 925 532 532
q20 321 330 218 218
q21 4139 2509 2327 2327
q22 1050 1027 976 976
Total cold run time: 115653 ms
Total hot run time: 33561 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5080 5032 5046 5032
q2 242 327 226 226
q3 2160 2632 2288 2288
q4 1409 1839 1351 1351
q5 4401 4389 4361 4361
q6 217 166 135 135
q7 2058 1951 1790 1790
q8 2645 2520 2501 2501
q9 7217 7196 7023 7023
q10 3044 3174 2727 2727
q11 589 519 507 507
q12 660 753 627 627
q13 3514 3896 3301 3301
q14 279 301 267 267
q15 543 499 491 491
q16 444 471 425 425
q17 1139 1486 1398 1398
q18 7692 7515 7440 7440
q19 786 768 840 768
q20 1980 2060 1869 1869
q21 4755 4532 4413 4413
q22 1077 1121 1038 1038
Total cold run time: 51931 ms
Total hot run time: 49978 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2781919915
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2781927482
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782483839
<details>
<summary>TPC-H: <b>Total hot run time: 34214 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5103a6e2860c8456865f2035cb58790c78c0954c, data reload: false
------ Round 1 ----------------------------------
q1 26063 5095 5006 5006
q2 2062 285 184 184
q3 10397 1236 675 675
q4 10211 1007 516 516
q5 7545 2373 2382 2373
q6 191 170 136 136
q7 951 755 621 621
q8 9343 1321 1103 1103
q9 6816 5168 5181 5168
q10 6807 2322 1884 1884
q11 473 272 266 266
q12 350 356 221 221
q13 17783 3680 3121 3121
q14 221 214 204 204
q15 544 484 501 484
q16 639 616 577 577
q17 586 877 358 358
q18 7541 7133 7063 7063
q19 1739 979 573 573
q20 331 338 233 233
q21 3943 3362 2465 2465
q22 1093 1024 983 983
Total cold run time: 115629 ms
Total hot run time: 34214 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5185 5123 5140 5123
q2 239 327 225 225
q3 2173 2658 2309 2309
q4 1448 1851 1552 1552
q5 4450 4388 4401 4388
q6 221 166 125 125
q7 1996 1945 1775 1775
q8 2592 2608 2576 2576
q9 7285 7238 7084 7084
q10 2998 3189 2745 2745
q11 579 520 490 490
q12 674 769 651 651
q13 3556 3872 3249 3249
q14 274 312 291 291
q15 542 489 465 465
q16 644 698 637 637
q17 1169 1570 1413 1413
q18 7771 7616 7374 7374
q19 809 876 1072 876
q20 2085 1974 1878 1878
q21 5125 4846 4821 4821
q22 1117 1094 1040 1040
Total cold run time: 52932 ms
Total hot run time: 51087 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2795756674
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2795974234
# BE UT Coverage Report
Increment line coverage `0.00% (0/22)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/a023e7737c776901c1893fd5eea3c26e314f3df8_a023e7737c776901c1893fd5eea3c26e314f3df8/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/a023e7737c776901c1893fd5eea3c26e314f3df8_a023e7737c776901c1893fd5eea3c26e314f3df8/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.46% (14106/26887) |
| Line Coverage | 41.27% (121966/295527) |
| Region Coverage | 40.03% (62114/155182) |
| Branch Coverage | 34.68% (31102/89690) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782530969
<details>
<summary>ClickBench: <b>Total hot run time: 30.91 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5103a6e2860c8456865f2035cb58790c78c0954c, data reload: false
query1 0.04 0.04 0.03
query2 0.12 0.10 0.11
query3 0.25 0.20 0.19
query4 1.58 0.20 0.20
query5 0.61 0.59 0.59
query6 1.20 0.72 0.73
query7 0.02 0.01 0.01
query8 0.04 0.03 0.03
query9 0.58 0.52 0.53
query10 0.59 0.60 0.55
query11 0.16 0.10 0.11
query12 0.14 0.11 0.12
query13 0.61 0.59 0.61
query14 2.81 2.70 2.70
query15 0.95 0.85 0.85
query16 0.40 0.38 0.37
query17 1.03 1.07 1.00
query18 0.21 0.20 0.20
query19 1.87 1.90 1.84
query20 0.01 0.01 0.01
query21 15.35 0.88 0.54
query22 0.76 1.09 0.65
query23 15.07 1.38 0.59
query24 7.01 2.02 0.50
query25 0.47 0.14 0.19
query26 0.60 0.16 0.13
query27 0.05 0.05 0.05
query28 9.50 0.85 0.43
query29 12.52 3.99 3.34
query30 0.24 0.09 0.06
query31 2.84 0.57 0.39
query32 3.23 0.55 0.47
query33 2.99 3.07 3.05
query34 15.65 5.13 4.52
query35 4.56 4.54 4.54
query36 0.67 0.50 0.48
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.02 0.02 0.02
query40 0.17 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 105.2 s
Total hot run time: 30.91 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782304287
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on code in PR #49822:
URL: https://github.com/apache/doris/pull/49822#discussion_r2084403862
##########
be/src/pipeline/exec/group_commit_scan_operator.cpp:
##########
@@ -46,8 +45,21 @@ Status GroupCommitLocalState::init(RuntimeState* state, LocalStateInfo& info) {
auto& p = _parent->cast<GroupCommitOperatorX>();
_get_block_dependency = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
"GroupCommitGetBlockDependency", true);
- return state->exec_env()->group_commit_mgr()->get_load_block_queue(
+ _timer_dependency = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
+ "GroupCommitTimerDependency", true);
+ auto st = state->exec_env()->group_commit_mgr()->get_load_block_queue(
p._table_id, state->fragment_instance_id(), load_block_queue, _get_block_dependency);
+ if (st.ok()) {
+ DCHECK(load_block_queue != nullptr);
+ _timer_dependency->block();
+ _runtime_filter_timer = std::make_shared<pipeline::RuntimeFilterTimer>(
+ MonotonicMillis(), load_block_queue->get_group_commit_interval_ms(),
+ _timer_dependency);
+ std::vector<std::shared_ptr<pipeline::RuntimeFilterTimer>> timers;
Review Comment:
group commit需要按照提交时间被唤醒，现在只有runtime filter支持这种
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #49822:
URL: https://github.com/apache/doris/pull/49822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #49822:
URL: https://github.com/apache/doris/pull/49822#discussion_r2062579949
##########
be/src/pipeline/exec/group_commit_scan_operator.cpp:
##########
@@ -46,8 +45,21 @@ Status GroupCommitLocalState::init(RuntimeState* state, LocalStateInfo& info) {
auto& p = _parent->cast<GroupCommitOperatorX>();
_get_block_dependency = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
"GroupCommitGetBlockDependency", true);
- return state->exec_env()->group_commit_mgr()->get_load_block_queue(
+ _timer_dependency = Dependency::create_shared(_parent->operator_id(), _parent->node_id(),
+ "GroupCommitTimerDependency", true);
+ auto st = state->exec_env()->group_commit_mgr()->get_load_block_queue(
p._table_id, state->fragment_instance_id(), load_block_queue, _get_block_dependency);
+ if (st.ok()) {
+ DCHECK(load_block_queue != nullptr);
+ _timer_dependency->block();
+ _runtime_filter_timer = std::make_shared<pipeline::RuntimeFilterTimer>(
+ MonotonicMillis(), load_block_queue->get_group_commit_interval_ms(),
+ _timer_dependency);
+ std::vector<std::shared_ptr<pipeline::RuntimeFilterTimer>> timers;
Review Comment:
为啥会用到runtime filter？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2795807383
<details>
<summary>ClickBench: <b>Total hot run time: 31.18 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a023e7737c776901c1893fd5eea3c26e314f3df8, data reload: false
query1 0.03 0.03 0.03
query2 0.13 0.10 0.10
query3 0.25 0.20 0.20
query4 1.59 0.19 0.20
query5 0.59 0.60 0.60
query6 1.19 0.72 0.72
query7 0.02 0.01 0.02
query8 0.04 0.03 0.03
query9 0.58 0.53 0.51
query10 0.59 0.59 0.57
query11 0.16 0.10 0.10
query12 0.15 0.12 0.11
query13 0.61 0.60 0.60
query14 2.81 2.79 2.69
query15 0.92 0.84 0.85
query16 0.38 0.38 0.39
query17 1.05 1.00 1.04
query18 0.20 0.19 0.19
query19 1.90 1.99 1.80
query20 0.01 0.01 0.01
query21 15.35 0.87 0.56
query22 0.74 1.14 0.69
query23 14.98 1.38 0.61
query24 6.89 1.25 0.85
query25 0.52 0.20 0.08
query26 0.58 0.16 0.14
query27 0.05 0.05 0.05
query28 9.10 0.94 0.43
query29 12.54 3.99 3.31
query30 0.24 0.09 0.06
query31 2.84 0.59 0.39
query32 3.23 0.54 0.47
query33 3.08 3.03 3.05
query34 16.17 5.14 4.54
query35 4.53 4.56 4.50
query36 0.66 0.51 0.49
query37 0.09 0.06 0.05
query38 0.06 0.04 0.03
query39 0.02 0.02 0.02
query40 0.18 0.14 0.13
query41 0.08 0.02 0.03
query42 0.04 0.03 0.02
query43 0.03 0.04 0.03
Total cold run time: 105.2 s
Total hot run time: 31.18 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2796146897
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2889629226
# BE UT Coverage Report
Increment line coverage `0.00% (0/22)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/18509efc38e4dc855667894f374bce30e7973650_18509efc38e4dc855667894f374bce30e7973650/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/18509efc38e4dc855667894f374bce30e7973650_18509efc38e4dc855667894f374bce30e7973650/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 55.81% (14886/26671) |
| Line Coverage | 44.60% (131908/295739) |
| Region Coverage | 43.73% (66413/151859) |
| Branch Coverage | 38.33% (34025/88762) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2892688607
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782133807
# BE UT Coverage Report
Increment line coverage `0.00% (0/22)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/3da47f3f6e3e421d67cd3d36a366ef6c65f64007_3da47f3f6e3e421d67cd3d36a366ef6c65f64007/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/3da47f3f6e3e421d67cd3d36a366ef6c65f64007_3da47f3f6e3e421d67cd3d36a366ef6c65f64007/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.23% (13998/26800) |
| Line Coverage | 40.97% (120615/294387) |
| Region Coverage | 39.69% (61349/154588) |
| Branch Coverage | 34.31% (30628/89274) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782516094
<details>
<summary>TPC-DS: <b>Total hot run time: 193538 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5103a6e2860c8456865f2035cb58790c78c0954c, data reload: false
query1 1382 1050 1048 1048
query2 6246 2000 1952 1952
query3 11128 4665 4881 4665
query4 25425 23714 23138 23138
query5 4899 636 445 445
query6 307 201 206 201
query7 3999 486 288 288
query8 302 233 214 214
query9 8506 2553 2563 2553
query10 490 318 252 252
query11 15219 15188 14972 14972
query12 159 103 105 103
query13 1540 505 390 390
query14 9087 6204 6118 6118
query15 202 181 166 166
query16 7244 638 481 481
query17 1160 749 600 600
query18 2006 427 341 341
query19 198 199 202 199
query20 130 131 121 121
query21 221 124 107 107
query22 4521 4865 4351 4351
query23 34453 33554 33361 33361
query24 8540 2411 2427 2411
query25 517 472 404 404
query26 1167 269 154 154
query27 2939 495 340 340
query28 4966 2449 2471 2449
query29 745 547 465 465
query30 287 237 202 202
query31 915 877 796 796
query32 72 62 58 58
query33 525 364 318 318
query34 798 896 519 519
query35 835 862 791 791
query36 972 1027 898 898
query37 114 102 75 75
query38 4189 4135 4260 4135
query39 1501 1437 1431 1431
query40 216 121 106 106
query41 51 54 52 52
query42 119 110 111 110
query43 527 517 500 500
query44 1298 813 800 800
query45 182 173 161 161
query46 842 1032 649 649
query47 1838 1905 1791 1791
query48 380 438 306 306
query49 753 532 405 405
query50 636 689 446 446
query51 4283 4263 4234 4234
query52 107 112 97 97
query53 241 261 192 192
query54 591 578 518 518
query55 82 78 85 78
query56 303 306 282 282
query57 1179 1214 1159 1159
query58 270 265 264 264
query59 2906 2812 2886 2812
query60 331 323 317 317
query61 154 152 157 152
query62 768 758 720 720
query63 236 191 189 189
query64 4278 1114 799 799
query65 4477 4315 4297 4297
query66 1532 430 342 342
query67 16194 15675 15342 15342
query68 8750 881 518 518
query69 486 303 265 265
query70 1204 1110 1108 1108
query71 471 310 301 301
query72 5788 4748 4934 4748
query73 737 661 347 347
query74 8898 8940 8974 8940
query75 3909 3221 2672 2672
query76 3719 1200 766 766
query77 798 369 294 294
query78 10007 10223 9310 9310
query79 1915 825 558 558
query80 586 508 514 508
query81 463 259 226 226
query82 451 122 97 97
query83 262 252 236 236
query84 245 101 82 82
query85 784 357 318 318
query86 334 301 290 290
query87 4468 4438 4375 4375
query88 3361 2213 2193 2193
query89 388 313 289 289
query90 1904 208 211 208
query91 142 141 115 115
query92 73 60 54 54
query93 1540 941 585 585
query94 678 431 318 318
query95 374 297 288 288
query96 477 566 278 278
query97 3166 3244 3120 3120
query98 233 212 206 206
query99 1415 1398 1293 1293
Total cold run time: 280706 ms
Total hot run time: 193538 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2781979813
<details>
<summary>TPC-H: <b>Total hot run time: 34367 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3da47f3f6e3e421d67cd3d36a366ef6c65f64007, data reload: false
------ Round 1 ----------------------------------
q1 25929 5066 5017 5017
q2 2072 273 187 187
q3 10398 1286 707 707
q4 10219 1003 561 561
q5 7521 2404 2367 2367
q6 188 161 133 133
q7 916 758 627 627
q8 9313 1323 1159 1159
q9 6905 5193 5208 5193
q10 6885 2329 1901 1901
q11 482 290 276 276
q12 358 367 223 223
q13 17786 3760 3028 3028
q14 232 229 208 208
q15 545 488 481 481
q16 645 602 556 556
q17 618 866 382 382
q18 7589 7266 7111 7111
q19 1295 965 566 566
q20 355 339 225 225
q21 4481 3446 2510 2510
q22 1071 1004 949 949
Total cold run time: 115803 ms
Total hot run time: 34367 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5107 5113 5093 5093
q2 239 329 239 239
q3 2195 2661 2335 2335
q4 1424 1851 1496 1496
q5 4717 4406 4407 4406
q6 217 168 125 125
q7 1954 1921 1727 1727
q8 2626 2649 2597 2597
q9 7293 7112 7170 7112
q10 2960 3149 2746 2746
q11 578 525 499 499
q12 716 790 635 635
q13 3536 3914 3371 3371
q14 281 292 288 288
q15 538 509 506 506
q16 681 673 669 669
q17 1170 1583 1390 1390
q18 7836 7713 7420 7420
q19 840 828 890 828
q20 1907 1995 1884 1884
q21 5458 4916 4809 4809
q22 1112 1053 1017 1017
Total cold run time: 53385 ms
Total hot run time: 51192 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2795802207
<details>
<summary>TPC-DS: <b>Total hot run time: 193284 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a023e7737c776901c1893fd5eea3c26e314f3df8, data reload: false
query1 1403 1081 1046 1046
query2 6161 1991 1974 1974
query3 11149 4678 4929 4678
query4 25327 23808 23140 23140
query5 4823 629 486 486
query6 316 192 199 192
query7 3994 498 276 276
query8 291 239 225 225
query9 8517 2608 2596 2596
query10 483 330 281 281
query11 15280 15213 14888 14888
query12 161 110 117 110
query13 1561 527 380 380
query14 9079 6288 6125 6125
query15 201 178 170 170
query16 7610 620 471 471
query17 1122 740 604 604
query18 2007 405 313 313
query19 201 195 172 172
query20 122 124 118 118
query21 203 125 117 117
query22 4351 4483 4297 4297
query23 34560 33471 33502 33471
query24 8682 2453 2424 2424
query25 511 480 399 399
query26 732 292 157 157
query27 3050 503 345 345
query28 4790 2499 2424 2424
query29 663 591 473 473
query30 284 226 202 202
query31 915 861 784 784
query32 74 67 65 65
query33 551 374 319 319
query34 819 886 519 519
query35 830 878 786 786
query36 973 1016 905 905
query37 132 103 85 85
query38 4278 4300 4151 4151
query39 1554 1461 1442 1442
query40 225 125 110 110
query41 54 58 55 55
query42 121 110 103 103
query43 524 525 510 510
query44 1368 821 829 821
query45 190 168 169 168
query46 833 1025 639 639
query47 1797 1918 1869 1869
query48 369 409 301 301
query49 756 507 437 437
query50 655 710 405 405
query51 4265 4360 4279 4279
query52 122 106 97 97
query53 236 271 192 192
query54 601 580 515 515
query55 84 81 84 81
query56 322 308 288 288
query57 1134 1193 1091 1091
query58 267 263 274 263
query59 2761 2755 2664 2664
query60 332 332 316 316
query61 134 127 156 127
query62 787 732 671 671
query63 225 191 183 183
query64 3078 1057 739 739
query65 4498 4343 4414 4343
query66 1015 423 309 309
query67 16176 15529 15353 15353
query68 9321 896 506 506
query69 490 297 270 270
query70 1213 1099 1135 1099
query71 491 331 299 299
query72 5315 4825 4905 4825
query73 747 647 349 349
query74 9205 9017 8692 8692
query75 4342 3200 2714 2714
query76 3719 1186 762 762
query77 829 391 279 279
query78 10015 10386 9343 9343
query79 1789 804 562 562
query80 634 514 434 434
query81 485 257 223 223
query82 437 125 100 100
query83 263 258 255 255
query84 243 108 93 93
query85 777 357 314 314
query86 333 297 280 280
query87 4378 4560 4392 4392
query88 3326 2247 2254 2247
query89 406 308 282 282
query90 1921 217 211 211
query91 148 148 113 113
query92 81 63 59 59
query93 1190 952 588 588
query94 662 391 311 311
query95 374 296 286 286
query96 491 568 272 272
query97 3163 3256 3073 3073
query98 231 206 204 204
query99 1458 1414 1308 1308
Total cold run time: 279166 ms
Total hot run time: 193284 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2781997212
<details>
<summary>ClickBench: <b>Total hot run time: 31.65 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3da47f3f6e3e421d67cd3d36a366ef6c65f64007, data reload: false
query1 0.04 0.04 0.03
query2 0.14 0.11 0.11
query3 0.26 0.20 0.20
query4 1.59 0.20 0.20
query5 0.60 0.56 0.58
query6 1.19 0.72 0.72
query7 0.02 0.02 0.02
query8 0.05 0.03 0.04
query9 0.57 0.51 0.50
query10 0.61 0.58 0.57
query11 0.15 0.11 0.11
query12 0.14 0.11 0.11
query13 0.61 0.58 0.61
query14 2.72 2.81 2.83
query15 0.94 0.84 0.84
query16 0.40 0.38 0.37
query17 1.03 1.01 1.01
query18 0.21 0.19 0.19
query19 1.88 1.97 1.80
query20 0.01 0.01 0.02
query21 15.37 0.91 0.54
query22 0.74 1.20 0.69
query23 14.93 1.35 0.61
query24 7.36 1.25 1.20
query25 0.49 0.23 0.08
query26 0.65 0.16 0.12
query27 0.05 0.05 0.04
query28 9.44 0.86 0.43
query29 12.56 4.08 3.41
query30 0.25 0.09 0.07
query31 2.83 0.61 0.38
query32 3.23 0.54 0.45
query33 3.09 3.10 3.10
query34 15.60 5.19 4.50
query35 4.49 4.45 4.53
query36 0.67 0.49 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.13 0.13
query41 0.09 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 105.41 s
Total hot run time: 31.65 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2889565419
<details>
<summary>TPC-DS: <b>Total hot run time: 192651 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18509efc38e4dc855667894f374bce30e7973650, data reload: false
query1 1402 1077 1059 1059
query2 6440 1862 1866 1862
query3 11119 4647 4514 4514
query4 24694 23314 22840 22840
query5 3717 618 463 463
query6 307 218 226 218
query7 3987 486 292 292
query8 308 256 246 246
query9 8519 2640 2650 2640
query10 446 322 283 283
query11 15268 15031 14791 14791
query12 172 109 108 108
query13 1567 542 405 405
query14 9160 6145 6118 6118
query15 248 187 171 171
query16 7592 658 498 498
query17 1195 721 584 584
query18 2026 413 322 322
query19 221 202 171 171
query20 135 127 122 122
query21 210 128 113 113
query22 4488 4653 4484 4484
query23 34511 33653 33807 33653
query24 8417 2456 2424 2424
query25 530 477 385 385
query26 1432 271 152 152
query27 2757 512 346 346
query28 4645 2200 2166 2166
query29 686 576 458 458
query30 273 225 194 194
query31 926 856 782 782
query32 71 61 63 61
query33 556 360 348 348
query34 826 896 541 541
query35 832 815 758 758
query36 1004 1016 881 881
query37 108 98 75 75
query38 4222 4165 4123 4123
query39 1532 1487 1438 1438
query40 203 122 108 108
query41 55 55 52 52
query42 124 111 113 111
query43 521 521 473 473
query44 1344 855 871 855
query45 181 172 168 168
query46 855 1038 662 662
query47 1882 1906 1859 1859
query48 405 430 330 330
query49 751 556 435 435
query50 674 704 406 406
query51 4269 4226 4313 4226
query52 121 108 96 96
query53 221 263 188 188
query54 598 584 537 537
query55 90 90 84 84
query56 323 324 300 300
query57 1218 1219 1164 1164
query58 281 268 265 265
query59 2654 2776 2755 2755
query60 349 329 315 315
query61 134 133 156 133
query62 794 716 678 678
query63 227 188 193 188
query64 3988 1049 715 715
query65 4452 4373 4332 4332
query66 1123 405 316 316
query67 16094 15532 15451 15451
query68 8259 884 553 553
query69 479 311 267 267
query70 1226 1145 1088 1088
query71 458 323 297 297
query72 5709 4763 4641 4641
query73 684 596 363 363
query74 9173 9139 8928 8928
query75 3899 3204 2664 2664
query76 3765 1188 738 738
query77 786 368 280 280
query78 10110 10171 9388 9388
query79 2303 816 582 582
query80 572 603 458 458
query81 490 253 213 213
query82 461 129 92 92
query83 248 244 235 235
query84 249 100 83 83
query85 802 367 313 313
query86 375 298 279 279
query87 4374 4417 4278 4278
query88 3831 2396 2358 2358
query89 418 314 283 283
query90 1853 211 207 207
query91 143 158 112 112
query92 75 62 53 53
query93 1789 954 613 613
query94 654 427 321 321
query95 365 285 271 271
query96 493 560 284 284
query97 2713 2822 2656 2656
query98 231 210 211 210
query99 1361 1373 1252 1252
Total cold run time: 278828 ms
Total hot run time: 192651 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2892688533
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2889500824
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2890784586
# BE Regression && UT Coverage Report
Increment line coverage `4.55% (1/22)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/49822_18509efc38e4dc855667894f374bce30e7973650_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/49822_18509efc38e4dc855667894f374bce30e7973650_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.00% (19420/26242) |
| Line Coverage | 62.59% (185076/295710) |
| Region Coverage | 60.62% (108141/178401) |
| Branch Coverage | 53.12% (53867/101398) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "mymeiyi (via GitHub)" <gi...@apache.org>.
mymeiyi commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2796614774
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2781991529
<details>
<summary>TPC-DS: <b>Total hot run time: 192238 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3da47f3f6e3e421d67cd3d36a366ef6c65f64007, data reload: false
query1 1363 1051 996 996
query2 6281 1930 1866 1866
query3 10984 4487 4664 4487
query4 55371 24123 22801 22801
query5 5064 525 453 453
query6 375 189 186 186
query7 5028 501 278 278
query8 326 255 244 244
query9 6712 2600 2614 2600
query10 432 321 259 259
query11 15207 15082 14803 14803
query12 157 114 106 106
query13 1144 508 392 392
query14 10867 6368 6222 6222
query15 191 202 175 175
query16 7130 648 514 514
query17 1077 743 583 583
query18 1560 420 335 335
query19 210 205 182 182
query20 128 118 130 118
query21 216 138 104 104
query22 4676 4454 4440 4440
query23 33936 33264 33338 33264
query24 5930 2417 2386 2386
query25 446 461 430 430
query26 726 266 147 147
query27 1948 476 344 344
query28 2964 2472 2491 2472
query29 572 565 428 428
query30 278 219 230 219
query31 865 897 800 800
query32 84 61 63 61
query33 445 368 319 319
query34 788 861 545 545
query35 816 839 808 808
query36 950 998 934 934
query37 123 106 81 81
query38 4254 4316 4021 4021
query39 1478 1414 1456 1414
query40 223 136 104 104
query41 53 55 50 50
query42 123 109 109 109
query43 476 516 487 487
query44 1365 848 816 816
query45 181 185 173 173
query46 843 1017 664 664
query47 1840 1905 1815 1815
query48 401 424 319 319
query49 696 518 420 420
query50 669 692 417 417
query51 4222 4320 4144 4144
query52 106 104 98 98
query53 233 270 194 194
query54 598 594 532 532
query55 88 81 90 81
query56 310 299 296 296
query57 1161 1188 1156 1156
query58 280 266 253 253
query59 2682 2888 2779 2779
query60 348 323 314 314
query61 131 131 129 129
query62 742 745 668 668
query63 230 188 192 188
query64 1802 1071 721 721
query65 4392 4274 4273 4273
query66 779 402 304 304
query67 16032 15933 15346 15346
query68 6828 892 517 517
query69 545 301 263 263
query70 1160 1069 1156 1069
query71 501 316 305 305
query72 5972 4792 4772 4772
query73 1269 631 357 357
query74 8980 9166 8882 8882
query75 3964 3213 2702 2702
query76 4261 1193 770 770
query77 642 358 276 276
query78 10222 10232 9207 9207
query79 1962 805 561 561
query80 569 526 445 445
query81 475 259 221 221
query82 330 122 102 102
query83 258 312 236 236
query84 297 106 85 85
query85 788 359 304 304
query86 368 307 289 289
query87 4393 4344 4444 4344
query88 3412 2187 2204 2187
query89 402 305 282 282
query90 1916 202 211 202
query91 143 143 111 111
query92 73 60 61 60
query93 1277 968 581 581
query94 634 398 319 319
query95 367 293 289 289
query96 492 553 273 273
query97 3153 3199 3097 3097
query98 229 206 204 204
query99 1333 1401 1259 1259
Total cold run time: 299323 ms
Total hot run time: 192238 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](group commit) reduce cpu cost for group_commit get_block [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49822:
URL: https://github.com/apache/doris/pull/49822#issuecomment-2782934939
# BE UT Coverage Report
Increment line coverage `0.00% (0/22)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/5103a6e2860c8456865f2035cb58790c78c0954c_5103a6e2860c8456865f2035cb58790c78c0954c/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/5103a6e2860c8456865f2035cb58790c78c0954c_5103a6e2860c8456865f2035cb58790c78c0954c/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.26% (14022/26832) |
| Line Coverage | 41.04% (121049/294956) |
| Region Coverage | 39.78% (61611/154882) |
| Branch Coverage | 34.45% (30844/89538) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org