You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "deardeng (via GitHub)" <gi...@apache.org> on 2025/12/16 12:13:01 UTC
[PR] [fix](cloud) Fix _balanced_tablets_shards leak memory and fid some cloud balance case [doris]
deardeng opened a new pull request, #59093:
URL: https://github.com/apache/doris/pull/59093
### What problem does this PR solve?
1. When fixing the issue of CloudWarmUpManager leaking a small amount of memory and failing to remove (tablet_id, peer_addr) from memory during warm-up balance when file cache meta is not retrieved from source, the issue was resolved.
2. Some cases that were fixing balance issues were affected by PR(https://github.com/apache/doris/pull/57770) optimizations for small files.
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3672230917
<details>
<summary>ClickBench: <b>Total hot run time: 27.35 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 005bf6e4178b3eaf5786ae75082cad2be585ae94, data reload: false
query1 0.06 0.05 0.04
query2 0.10 0.05 0.05
query3 0.26 0.09 0.09
query4 1.61 0.11 0.12
query5 0.28 0.26 0.25
query6 1.16 0.65 0.64
query7 0.03 0.03 0.03
query8 0.05 0.05 0.04
query9 0.58 0.51 0.51
query10 0.55 0.56 0.55
query11 0.16 0.11 0.12
query12 0.21 0.11 0.12
query13 0.62 0.62 0.61
query14 1.00 1.00 0.97
query15 0.81 0.80 0.81
query16 0.41 0.39 0.43
query17 1.03 1.05 1.06
query18 0.24 0.22 0.22
query19 1.91 1.85 1.74
query20 0.02 0.01 0.02
query21 15.47 0.31 0.15
query22 4.88 0.05 0.05
query23 16.12 0.29 0.10
query24 0.99 0.48 0.19
query25 0.12 0.06 0.08
query26 0.15 0.13 0.13
query27 0.08 0.06 0.04
query28 3.34 1.23 1.03
query29 12.57 4.05 3.23
query30 0.28 0.14 0.12
query31 2.82 0.63 0.40
query32 3.24 0.55 0.46
query33 3.07 3.05 3.05
query34 16.84 5.22 4.62
query35 4.48 4.55 4.61
query36 0.67 0.50 0.49
query37 0.12 0.07 0.06
query38 0.07 0.04 0.04
query39 0.04 0.04 0.03
query40 0.18 0.14 0.14
query41 0.09 0.04 0.03
query42 0.05 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 96.8 s
Total hot run time: 27.35 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3672552037
# BE UT Coverage Report
Increment line coverage `0.00% (0/26)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/005bf6e4178b3eaf5786ae75082cad2be585ae94_005bf6e4178b3eaf5786ae75082cad2be585ae94/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/005bf6e4178b3eaf5786ae75082cad2be585ae94_005bf6e4178b3eaf5786ae75082cad2be585ae94/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.40% (18850/35297) |
| Line Coverage | 39.24% (174730/445275) |
| Region Coverage | 33.79% (135237/400233) |
| Branch Coverage | 34.71% (58280/167922) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3668046783
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3672728487
# BE Regression && UT Coverage Report
Increment line coverage `0.00% (0/26)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/59093_005bf6e4178b3eaf5786ae75082cad2be585ae94_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/59093_005bf6e4178b3eaf5786ae75082cad2be585ae94_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.26% (24991/34586) |
| Line Coverage | 58.99% (262262/444594) |
| Region Coverage | 53.86% (218026/404783) |
| Branch Coverage | 55.41% (93481/168694) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3672213185
<details>
<summary>TPC-DS: <b>Total hot run time: 178393 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 005bf6e4178b3eaf5786ae75082cad2be585ae94, data reload: false
query5 5317 604 440 440
query6 337 231 219 219
query7 4234 454 278 278
query8 293 250 241 241
query9 8756 2557 2523 2523
query10 552 379 325 325
query11 15389 14782 14988 14782
query12 186 117 114 114
query13 1291 520 415 415
query14 6429 3033 2739 2739
query14_1 2661 2630 2653 2630
query15 215 202 179 179
query16 879 492 457 457
query17 1151 733 609 609
query18 2724 449 358 358
query19 235 234 213 213
query20 142 117 112 112
query21 220 142 117 117
query22 4075 4147 3956 3956
query23 16858 16139 15863 15863
query23_1 16152 16020 15954 15954
query24 7367 1663 1250 1250
query24_1 1225 1247 1257 1247
query25 609 509 460 460
query26 1281 275 162 162
query27 2724 468 318 318
query28 4443 2144 2135 2135
query29 811 568 471 471
query30 325 251 221 221
query31 819 710 628 628
query32 87 75 70 70
query33 554 351 308 308
query34 896 892 531 531
query35 828 828 734 734
query36 878 894 840 840
query37 135 93 77 77
query38 2807 2834 2791 2791
query39 786 743 734 734
query39_1 692 703 701 701
query40 228 142 125 125
query41 70 68 65 65
query42 106 104 105 104
query43 418 425 407 407
query44 1327 748 745 745
query45 199 192 182 182
query46 873 976 606 606
query47 1680 1749 1643 1643
query48 333 326 247 247
query49 642 461 350 350
query50 660 297 223 223
query51 3826 3900 3887 3887
query52 103 109 99 99
query53 319 346 292 292
query54 284 265 247 247
query55 79 78 70 70
query56 289 301 298 298
query57 1143 1139 1100 1100
query58 332 248 258 248
query59 2441 2623 2405 2405
query60 318 316 287 287
query61 167 164 160 160
query62 717 671 624 624
query63 324 296 303 296
query64 4880 1368 1033 1033
query65 3998 3988 3959 3959
query66 1378 432 317 317
query67 15386 15195 14936 14936
query68 8257 1012 725 725
query69 503 350 309 309
query70 1079 977 991 977
query71 416 308 280 280
query72 6040 4917 4949 4917
query73 657 560 305 305
query74 8753 8735 8569 8569
query75 3233 3163 2808 2808
query76 3932 1135 716 716
query77 592 399 291 291
query78 9463 9676 8803 8803
query79 1692 866 609 609
query80 742 651 562 562
query81 517 281 227 227
query82 208 133 101 101
query83 264 253 240 240
query84 267 122 108 108
query85 941 539 470 470
query86 376 295 292 292
query87 3047 3021 2915 2915
query88 3236 2277 2272 2272
query89 466 422 388 388
query90 2215 161 157 157
query91 175 166 146 146
query92 98 67 62 62
query93 1876 923 569 569
query94 501 309 285 285
query95 570 379 318 318
query96 590 476 211 211
query97 2243 2298 2241 2241
query98 215 203 198 198
query99 1262 1293 1218 1218
Total cold run time: 262424 ms
Total hot run time: 178393 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #59093:
URL: https://github.com/apache/doris/pull/59093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3671802807
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix _balanced_tablets_shards leak memory and fid some cloud balance case [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3660219399
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #59093:
URL: https://github.com/apache/doris/pull/59093#discussion_r2629261990
##########
be/src/cloud/cloud_warm_up_manager.h:
##########
@@ -98,6 +98,8 @@ class CloudWarmUpManager {
std::unordered_map<int64_t, std::pair<std::string, int32_t>> get_all_balanced_tablets() const;
private:
+ void schedule_remove_balanced_tablet(int64_t tablet_id);
+ static void clean_up_expired_mappings(void* arg);
Review Comment:
why do we expose this static function from .cpp file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3668046861
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix balanced_tablets_shards leak memory and fix some file cache case [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59093:
URL: https://github.com/apache/doris/pull/59093#issuecomment-3672176498
<details>
<summary>TPC-H: <b>Total hot run time: 36481 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 005bf6e4178b3eaf5786ae75082cad2be585ae94, data reload: false
------ Round 1 ----------------------------------
q1 17617 4374 4060 4060
q2 2034 347 238 238
q3 10206 1334 753 753
q4 10207 833 316 316
q5 7462 2309 1989 1989
q6 196 172 134 134
q7 1011 872 718 718
q8 9362 1480 1130 1130
q9 7037 5376 5425 5376
q10 6843 2409 1991 1991
q11 530 326 300 300
q12 702 756 588 588
q13 17818 3724 3054 3054
q14 287 295 277 277
q15 596 511 516 511
q16 715 683 620 620
q17 715 869 447 447
q18 7701 7829 7964 7829
q19 1152 1023 659 659
q20 438 381 252 252
q21 4808 4272 4191 4191
q22 1113 1067 1048 1048
Total cold run time: 108550 ms
Total hot run time: 36481 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4263 4598 4440 4440
q2 322 408 322 322
q3 2217 2980 2496 2496
q4 1435 1891 1685 1685
q5 4530 4508 4573 4508
q6 217 169 123 123
q7 1987 1963 1814 1814
q8 2626 2534 2523 2523
q9 7629 7513 7546 7513
q10 3097 3218 2923 2923
q11 584 514 496 496
q12 697 724 571 571
q13 3272 3679 3010 3010
q14 266 291 252 252
q15 534 494 489 489
q16 622 628 615 615
q17 1098 1342 1331 1331
q18 7144 6999 7072 6999
q19 849 811 832 811
q20 1919 1962 1819 1819
q21 4628 4277 4059 4059
q22 1136 1021 956 956
Total cold run time: 51072 ms
Total hot run time: 49755 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org