You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2025/04/08 12:11:28 UTC
[PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Yukang-Lian opened a new pull request, #49883:
URL: https://github.com/apache/doris/pull/49883
Pick #48759
This PR introduces a delay strategy for cumulative compaction tasks that protects the thread pool from becoming overwhelmed by resource-intensive operations.
The new `should_delay_submission()` function implements a delay strategy that:
- Only activates when the thread pool size meets a configurable minimum threshold
- Checks if the thread pool is near capacity (0-1 threads remaining)
- Evaluates whether the compaction task is resource-intensive based on input size and row count
When these conditions are met, the function returns true to indicate the submission should be delayed, preventing potential thread pool exhaustion and system performance degradation.
The implementation includes three configurable thresholds:
- `min_threads_for_cumu_delay_strategy`: Minimum thread pool size to activate the strategy
- `cumu_delay_strategy_size`: Maximum input size threshold (in bytes)
- `cumu_delay_strategy_row_num`: Maximum input row count threshold
Appropriate warning logs are generated when a resource-intensive task is delayed, including the tablet ID and resource metrics for easier troubleshooting.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2786232380
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2789468331
# BE UT Coverage Report
Increment line coverage `22.07% (32/145)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/143addd27154328fec97055df56c2b3cde1586a3_143addd27154328fec97055df56c2b3cde1586a3/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/143addd27154328fec97055df56c2b3cde1586a3_143addd27154328fec97055df56c2b3cde1586a3/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 40.04% (10490/26202) |
| Line Coverage | 30.72% (88240/287209) |
| Region Coverage | 29.83% (45450/152367) |
| Branch Coverage | 26.20% (23039/87948) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2786594438
<details>
<summary>TPC-DS: <b>Total hot run time: 197065 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 143addd27154328fec97055df56c2b3cde1586a3, data reload: false
query1 1333 924 933 924
query2 6232 2021 1997 1997
query3 10826 4273 4192 4192
query4 66126 28890 23347 23347
query5 5039 451 456 451
query6 428 178 175 175
query7 5675 311 307 307
query8 312 209 216 209
query9 9261 2672 2621 2621
query10 474 285 281 281
query11 17916 15162 15619 15162
query12 161 105 108 105
query13 1566 464 439 439
query14 10575 6776 7747 6776
query15 203 196 182 182
query16 7103 528 514 514
query17 1086 581 569 569
query18 1988 324 317 317
query19 220 152 170 152
query20 120 108 106 106
query21 218 104 109 104
query22 4775 4533 4511 4511
query23 34791 34186 33956 33956
query24 6059 2881 2859 2859
query25 510 430 454 430
query26 656 172 170 170
query27 1840 358 362 358
query28 4216 2506 2471 2471
query29 726 467 447 447
query30 249 166 170 166
query31 978 846 849 846
query32 73 59 60 59
query33 468 321 304 304
query34 911 512 532 512
query35 847 747 740 740
query36 1097 987 976 976
query37 123 67 73 67
query38 4136 4063 4016 4016
query39 1561 1497 1460 1460
query40 213 105 105 105
query41 51 52 48 48
query42 121 101 106 101
query43 536 503 486 486
query44 1177 844 838 838
query45 188 172 173 172
query46 1184 730 729 729
query47 2005 1960 1935 1935
query48 504 407 413 407
query49 735 418 430 418
query50 871 441 440 440
query51 7364 7251 7000 7000
query52 102 94 92 92
query53 263 188 186 186
query54 588 460 463 460
query55 84 79 80 79
query56 256 249 260 249
query57 1241 1129 1090 1090
query58 234 215 218 215
query59 3081 2874 3093 2874
query60 281 257 249 249
query61 109 117 132 117
query62 788 678 667 667
query63 220 196 201 196
query64 1387 663 636 636
query65 3261 3199 3196 3196
query66 718 301 303 301
query67 15688 15605 15631 15605
query68 4459 586 574 574
query69 439 279 268 268
query70 1211 1017 1118 1017
query71 362 267 262 262
query72 6482 4395 4071 4071
query73 759 353 361 353
query74 10437 9103 9325 9103
query75 3386 2675 2688 2675
query76 1889 1073 1064 1064
query77 485 296 275 275
query78 10393 9608 9632 9608
query79 2142 610 645 610
query80 1280 437 436 436
query81 533 238 242 238
query82 490 91 88 88
query83 193 145 143 143
query84 278 82 70 70
query85 981 301 298 298
query86 422 312 308 308
query87 4410 4297 4317 4297
query88 4049 2429 2494 2429
query89 417 294 290 290
query90 2001 189 194 189
query91 190 148 155 148
query92 61 48 51 48
query93 2557 569 551 551
query94 928 299 283 283
query95 359 260 264 260
query96 618 281 285 281
query97 3334 3182 3197 3182
query98 216 204 201 201
query99 1535 1341 1276 1276
Total cold run time: 321424 ms
Total hot run time: 197065 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2786231717
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2786541065
<details>
<summary>TPC-H: <b>Total hot run time: 40392 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 143addd27154328fec97055df56c2b3cde1586a3, data reload: false
------ Round 1 ----------------------------------
q1 17570 6828 6693 6693
q2 2079 176 175 175
q3 10611 1071 1111 1071
q4 10567 775 759 759
q5 7755 2877 2826 2826
q6 224 136 136 136
q7 970 621 609 609
q8 9361 1970 2048 1970
q9 6646 6430 6476 6430
q10 7001 2283 2307 2283
q11 463 259 255 255
q12 398 220 206 206
q13 17764 3005 3030 3005
q14 252 211 224 211
q15 525 464 465 464
q16 670 576 579 576
q17 986 585 578 578
q18 7497 6701 6746 6701
q19 1456 1107 1141 1107
q20 481 206 207 206
q21 4050 3236 3154 3154
q22 1086 977 982 977
Total cold run time: 108412 ms
Total hot run time: 40392 ms
----- Round 2, with runtime_filter_mode=off -----
q1 6647 6668 6604 6604
q2 331 232 236 232
q3 2894 2799 2947 2799
q4 2057 1761 1765 1761
q5 5807 5784 5824 5784
q6 214 131 131 131
q7 2231 1783 1818 1783
q8 3401 3589 3589 3589
q9 8818 8920 8954 8920
q10 3582 3569 3516 3516
q11 588 489 502 489
q12 813 601 590 590
q13 9369 3166 3221 3166
q14 325 271 268 268
q15 523 474 456 456
q16 697 659 657 657
q17 1886 1603 1630 1603
q18 8288 7795 7656 7656
q19 1693 1669 1583 1583
q20 2111 1899 1830 1830
q21 5586 5375 5590 5375
q22 1124 1040 1015 1015
Total cold run time: 68985 ms
Total hot run time: 59807 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2788769096
run be_ut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #49883:
URL: https://github.com/apache/doris/pull/49883
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2786612157
<details>
<summary>ClickBench: <b>Total hot run time: 31.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 143addd27154328fec97055df56c2b3cde1586a3, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.02
query3 0.23 0.07 0.07
query4 1.62 0.10 0.10
query5 0.52 0.49 0.51
query6 1.14 0.72 0.72
query7 0.02 0.02 0.01
query8 0.04 0.02 0.03
query9 0.56 0.51 0.50
query10 0.56 0.54 0.55
query11 0.14 0.10 0.11
query12 0.15 0.11 0.11
query13 0.62 0.60 0.61
query14 2.73 2.88 2.74
query15 0.90 0.82 0.83
query16 0.37 0.38 0.38
query17 1.06 1.06 1.01
query18 0.23 0.22 0.22
query19 1.85 1.81 2.02
query20 0.01 0.01 0.02
query21 15.36 0.60 0.59
query22 2.42 2.13 1.85
query23 16.91 0.90 0.98
query24 3.45 0.53 0.33
query25 0.26 0.15 0.05
query26 0.25 0.14 0.14
query27 0.05 0.04 0.04
query28 11.23 0.49 0.46
query29 12.60 3.23 3.21
query30 0.26 0.07 0.06
query31 2.83 0.39 0.38
query32 3.27 0.46 0.46
query33 3.00 3.08 3.02
query34 17.34 4.55 4.51
query35 4.50 4.50 4.47
query36 0.67 0.51 0.47
query37 0.09 0.06 0.06
query38 0.05 0.03 0.03
query39 0.03 0.03 0.02
query40 0.16 0.12 0.13
query41 0.08 0.02 0.02
query42 0.04 0.02 0.02
query43 0.03 0.03 0.03
Total cold run time: 107.73 s
Total hot run time: 31.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Cherry-pick](branch-3.0) Pick "[Enhancement](Compaction) Add cumulative compaction delay strategy (#48759)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #49883:
URL: https://github.com/apache/doris/pull/49883#issuecomment-2789095780
run beut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org