You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2026/10/20 06:52:18 UTC
[PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
bobhan1 opened a new pull request, #69325:
URL: https://github.com/apache/doris/pull/69325
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Event-driven warmup can register a rowset before the target synchronizes its metadata. If the source compacts that rowset before the next synchronization, it never enters the target's local Rowset GC lifecycle, leaving its completed warmup record allocated indefinitely.
Reuse the existing tablet vacuum task to reclaim completed records that are absent from both visible and stale rowsets after a configurable grace period (one hour by default). Include tablets without stale or unused rowsets, and release oversized hash buckets after reclamation. Retain running attempts and records belonging to local query version paths.
Assign a generation to each accepted attempt so delayed callbacks and index registration cannot update a replacement record. Protect unused-Rowset state removal with the same metadata lock used by callbacks, preserving meta-before-GC lock order, and make the state getter read-only.
### Release note
Cloud backends reclaim orphaned completed rowset warmup records to reduce memory growth. The retention grace period is controlled by `warm_up_rowset_state_gc_grace_period_seconds`, which defaults to 3600 seconds.
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test: ASAN build and focused tests are running with `./run-be-ut.sh --run --filter='CloudTabletWarmUpStateTest.*:CloudTabletMgrTest.*:*PreferCache*:*FreshnessTolerance*' -j100`.
- [x] Manual test: clang-format 16 check, build header hygiene checks, and `git diff --check`.
- Behavior changed:
- [x] Yes. Aged orphaned completed warmup records are reclaimed and may subsequently be warmed again. Running and local visible/stale records remain tracked.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6292153632
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/69325_35dc9b00bddcf31a1fa5459ecda2738908103b75_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/69325_35dc9b00bddcf31a1fa5459ecda2738908103b75_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 77.28% (35878/46428) |
| Line Coverage | 62.51% (404830/647584) |
| Region Coverage | 58.97% (341803/579591) |
| Branch Coverage | 59.95% (158133/263776) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6286534501
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #69325:
URL: https://github.com/apache/doris/pull/69325#discussion_r4322480869
##########
be/src/cloud/cloud_tablet.cpp:
##########
@@ -688,6 +689,8 @@ void CloudTablet::remove_unused_rowsets() {
int64_t removed_delete_bitmap_num = 0;
OlapStopWatch watch;
{
+ // Warmup callbacks use _meta_lock; take it before _gc_mutex, as add_rowsets does.
+ std::unique_lock meta_lock(_meta_lock);
Review Comment:
[P2] Clear unused rowset caches after releasing the metadata lock
After a large compaction, the vacuum thread can reclaim many cached input segments in one pass. This new exclusive `_meta_lock` stays held while `rs->clear_cache()` clears every rowset in that batch; that function reaches `BlockFileCache::remove_if_cached_async()`, which synchronously walks/removes cache metadata for every block despite its name. `capture_rs_readers()`, rowset synchronization and warmup callbacks all need `_meta_lock`, so they now wait for the entire cache-cleanup batch. Please keep the protected state removal/selection here, then clear the selected rowsets' caches after releasing `_meta_lock` and `_gc_mutex`; `removed_rowsets` already keeps them alive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6291168655
<details>
<summary>TPC-DS: <b>Total hot run time: 150646 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35dc9b00bddcf31a1fa5459ecda2738908103b75, data reload: false
query5 4303 580 448 448
query6 435 206 193 193
query7 4797 431 223 223
query8 325 164 154 154
query9 8762 3093 3108 3093
query10 466 312 271 271
query11 6031 3849 3566 3566
query12 145 95 89 89
query13 1256 450 322 322
query14 6118 3907 3643 3643
query14_1 3414 3424 3405 3405
query15 205 202 182 182
query16 946 440 372 372
query17 848 664 544 544
query18 2417 433 310 310
query19 196 166 136 136
query20 86 86 82 82
query21 214 125 107 107
query22 12905 12984 12652 12652
query23 13496 12996 12589 12589
query23_1 12642 12520 12535 12520
query24 7409 812 469 469
query24_1 475 472 465 465
query25 542 429 371 371
query26 1273 251 138 138
query27 2783 445 278 278
query28 4641 1978 1983 1978
query29 1557 598 461 461
query30 302 227 180 180
query31 1018 943 800 800
query32 144 89 92 89
query33 524 302 253 253
query34 913 864 518 518
query35 739 770 642 642
query36 848 819 756 756
query37 125 99 80 80
query38 1774 1745 1650 1650
query39 779 784 747 747
query39_1 736 736 741 736
query40 223 123 96 96
query41 73 68 68 68
query42 87 84 87 84
query43 341 366 312 312
query44 1309 686 691 686
query45 184 188 197 188
query46 827 969 559 559
query47 2885 2976 2622 2622
query48 291 307 222 222
query49 565 393 291 291
query50 669 270 214 214
query51 10390 10425 10471 10425
query52 76 76 69 69
query53 181 204 155 155
query54 236 188 209 188
query55 72 64 60 60
query56 234 209 219 209
query57 1659 1623 1578 1578
query58 293 267 259 259
query59 2119 2187 1974 1974
query60 273 227 219 219
query61 144 147 147 147
query62 385 328 277 277
query63 189 156 159 156
query64 2717 896 728 728
query65 3346 3277 3280 3277
query66 1759 423 346 346
query67 20090 20122 19919 19919
query68 3063 992 618 618
query69 372 297 233 233
query70 841 776 789 776
query71 295 228 203 203
query72 2642 2600 2268 2268
query73 519 535 296 296
query74 4602 4524 4352 4352
query75 2343 2289 1921 1921
query76 2319 1020 579 579
query77 334 378 260 260
query78 9815 9827 9189 9189
query79 959 877 510 510
query80 784 419 370 370
query81 560 322 270 270
query82 570 129 102 102
query83 276 198 175 175
query84 303 124 105 105
query85 952 429 366 366
query86 353 236 228 228
query87 1993 1970 1802 1802
query88 3658 2677 2650 2650
query89 330 279 247 247
query90 1706 167 164 164
query91 161 153 129 129
query92 96 79 90 79
query93 1000 983 567 567
query94 496 292 262 262
query95 567 380 314 314
query96 663 481 233 233
query97 2452 2439 2361 2361
query98 162 152 147 147
query99 768 770 662 662
Total cold run time: 233820 ms
Total hot run time: 150646 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6285197297
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6290991533
<details>
<summary>TPC-H: <b>Total hot run time: 28929 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35dc9b00bddcf31a1fa5459ecda2738908103b75, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17663 5223 5155 5155
q2 2108 309 274 274
q3 10226 1380 832 832
q4 4674 463 337 337
q5 7451 801 530 530
q6 189 189 155 155
q7 730 795 611 611
q8 9266 1289 1150 1150
q9 5560 4212 4204 4204
q10 6849 1294 1030 1030
q11 434 246 225 225
q12 641 402 279 279
q13 18105 4464 3014 3014
q14 284 272 248 248
q15 q16 736 732 664 664
q17 1206 749 566 566
q18 6982 6189 6129 6129
q19 1274 962 641 641
q20 397 349 228 228
q21 5653 2393 2473 2393
q22 378 343 264 264
Total cold run time: 100806 ms
Total hot run time: 28929 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 5965 5981 6030 5981
q2 740 592 537 537
q3 5585 5901 5234 5234
q4 2062 2147 1690 1690
q5 5724 5734 5692 5692
q6 247 197 151 151
q7 2244 2001 1802 1802
q8 3008 2678 2712 2678
q9 7614 7485 7502 7485
q10 3686 3592 3166 3166
q11 584 407 396 396
q12 660 659 507 507
q13 3262 3611 3037 3037
q14 293 281 261 261
q15 q16 651 693 626 626
q17 7781 7206 6911 6911
q18 12955 12178 12861 12178
q19 923 816 822 816
q20 2135 2126 1933 1933
q21 5682 4572 4658 4572
q22 512 426 400 400
Total cold run time: 72313 ms
Total hot run time: 66053 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6286536849
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #69325:
URL: https://github.com/apache/doris/pull/69325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6285441850
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6291259460
<details>
<summary>ClickBench: <b>Total hot run time: 24.43 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35dc9b00bddcf31a1fa5459ecda2738908103b75, data reload: false
query1 0.01 0.01 0.00
query2 0.09 0.05 0.04
query3 0.26 0.09 0.08
query4 1.61 0.10 0.10
query5 0.26 0.24 0.24
query6 1.17 0.69 0.68
query7 0.04 0.01 0.00
query8 0.06 0.04 0.04
query9 0.41 0.34 0.35
query10 0.56 0.59 0.58
query11 0.19 0.13 0.15
query12 0.17 0.15 0.13
query13 0.50 0.49 0.50
query14 1.03 1.01 0.99
query15 0.64 0.60 0.60
query16 0.34 0.35 0.33
query17 1.12 1.13 1.13
query18 0.20 0.20 0.19
query19 2.07 2.02 2.01
query20 0.02 0.01 0.02
query21 15.48 0.18 0.13
query22 5.02 0.06 0.05
query23 16.27 0.33 0.11
query24 2.89 0.60 0.42
query25 0.12 0.05 0.04
query26 1.25 0.21 0.16
query27 0.05 0.03 0.04
query28 3.65 0.73 0.35
query29 12.55 4.25 3.36
query30 0.28 0.13 0.10
query31 2.77 0.57 0.32
query32 3.21 0.60 0.50
query33 3.14 3.22 3.18
query34 15.50 4.01 3.41
query35 3.36 3.37 3.36
query36 0.55 0.44 0.43
query37 0.09 0.06 0.07
query38 0.05 0.04 0.04
query39 0.04 0.03 0.03
query40 0.17 0.15 0.15
query41 0.09 0.03 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.03
Total cold run time: 97.36 s
Total hot run time: 24.43 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6315036435
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Reclaim orphaned rowset warmup states [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #69325:
URL: https://github.com/apache/doris/pull/69325#issuecomment-6315035743
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org