You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2026/02/25 11:35:06 UTC
[PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
bobhan1 opened a new pull request, #60832:
URL: https://github.com/apache/doris/pull/60832
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Fix a race condition in `CloudTabletMgr::get_tablet()` introduced by #57922 (commit `0918952c70`) that causes tablets to permanently disappear from `_tablet_map`, making them invisible to the compaction scheduler. This leads to MOW (Merge-on-Write) tables accumulating
hundreds of rowsets without any compaction under high-frequency import.
## Root Cause
Commit `0918952c70` refactored `get_tablet()` by moving `_cache->insert()` and `_tablet_map->put()` from inside the `SingleFlight` lambda to outside it. This introduced a race condition:
1. When N concurrent `get_tablet()` calls arrive for the same `tablet_id`, `SingleFlight` executes the load lambda only once (by the "leader"), but all N callers receive a `shared_ptr` pointing to the same `CloudTablet` object.
2. After `SingleFlight::load()` returns, all N callers independently execute:
```cpp
_cache->insert(key, value, ...) // each creates a competing LRU cache entry
_tablet_map->put(tablet) // each inserts into tablet_map
```
3. Each `_cache->insert()` evicts the previous entry for the same key. The evicted `Value`'s destructor calls `_tablet_map.erase(tablet.get())`. The safety check `it->second.get() == tablet` was designed to prevent erasing a newer tablet object — but here all callers
share the same raw pointer from `SingleFlight`, so the check always passes, and the erase succeeds.
4. After the last caller's old cache handle is released (when its returned `shared_ptr` goes out of scope), the destructor erases the entry from `_tablet_map`.
5. Crucially, subsequent `get_tablet()` calls find the tablet in the LRU cache (cache hit path), which never touches `_tablet_map`. So the tablet is permanently invisible to `_tablet_map` and can never re-enter it.
6. The compaction scheduler uses `get_weak_tablets()` which iterates `_tablet_map`, so it never sees these tablets and never schedules compaction for them.
## Before (original correct code, prior to #57922):
```cpp
auto load_tablet = [this, &key, ...](int64_t tablet_id) {
// load from meta service...
// Cache insert + tablet_map put INSIDE lambda — only leader executes
auto* handle = _cache->insert(key, value.release(), ...);
_tablet_map->put(std::move(tablet));
return ret;
};
s_singleflight_load_tablet.load(tablet_id, std::move(load_tablet));
```
## After #57922 (buggy code):
```cpp
auto load_tablet = [this, ...](int64_t tablet_id) {
// load from meta service...
return tablet; // just return raw tablet
};
auto result = s_singleflight_load_tablet.load(tablet_id, std::move(load_tablet));
// Cache insert + tablet_map put OUTSIDE lambda — ALL concurrent callers execute
_cache->insert(key, value.release(), ...);
_tablet_map->put(std::move(tablet));
```
## Fix
Move `_cache->insert()` and `_tablet_map->put()` back inside the `SingleFlight` lambda, ensuring only the leader caller performs cache insertion and `_tablet_map` registration. This restores the invariant that a single `get_tablet()` cache miss produces exactly one
LRU cache entry and one `_tablet_map` entry, eliminating the race condition.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3959689299
# BE Regression && UT Coverage Report
Increment line coverage `83.33% (15/18)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/60832_776316a720e1c0cc1016d631758b9ae78aaae291_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/60832_776316a720e1c0cc1016d631758b9ae78aaae291_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.56% (26135/36524) |
| Line Coverage | 54.27% (273530/504001) |
| Region Coverage | 51.58% (227333/440707) |
| Branch Coverage | 53.03% (97794/184400) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3958651287
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3958706884
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3958908443
<details>
<summary>TPC-H: <b>Total hot run time: 29169 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 776316a720e1c0cc1016d631758b9ae78aaae291, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17648 4642 4337 4337
q2 q3 10650 773 517 517
q4 4684 351 259 259
q5 7558 1192 1032 1032
q6 175 175 146 146
q7 774 854 660 660
q8 9294 1454 1335 1335
q9 4931 4804 4758 4758
q10 6824 1883 1630 1630
q11 451 270 247 247
q12 696 573 512 512
q13 17806 4239 3446 3446
q14 227 229 214 214
q15 966 798 787 787
q16 743 709 659 659
q17 690 830 443 443
q18 6018 5440 5395 5395
q19 1240 993 664 664
q20 553 507 414 414
q21 4652 1851 1475 1475
q22 339 295 239 239
Total cold run time: 96919 ms
Total hot run time: 29169 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 4475 4388 4360 4360
q2 q3 1766 2195 1716 1716
q4 845 1167 747 747
q5 4036 4306 4329 4306
q6 180 176 142 142
q7 1740 1594 1477 1477
q8 2406 2666 2500 2500
q9 7565 7302 7459 7302
q10 2703 2820 2391 2391
q11 531 441 413 413
q12 531 610 461 461
q13 4109 4517 3578 3578
q14 287 303 272 272
q15 833 808 820 808
q16 721 766 710 710
q17 1186 1597 1302 1302
q18 7093 6932 6660 6660
q19 974 993 935 935
q20 2107 2143 2020 2020
q21 4092 3578 3461 3461
q22 458 449 367 367
Total cold run time: 48638 ms
Total hot run time: 45928 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3959110266
# BE UT Coverage Report
Increment line coverage `11.11% (2/18)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/776316a720e1c0cc1016d631758b9ae78aaae291_776316a720e1c0cc1016d631758b9ae78aaae291/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/776316a720e1c0cc1016d631758b9ae78aaae291_776316a720e1c0cc1016d631758b9ae78aaae291/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.55% (19597/37291) |
| Line Coverage | 36.14% (182689/505501) |
| Region Coverage | 32.48% (141799/436539) |
| Branch Coverage | 33.44% (61459/183814) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3958968384
<details>
<summary>TPC-DS: <b>Total hot run time: 184647 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 776316a720e1c0cc1016d631758b9ae78aaae291, data reload: false
query5 4834 641 532 532
query6 334 228 211 211
query7 4236 465 289 289
query8 328 240 224 224
query9 8752 2756 2687 2687
query10 522 359 353 353
query11 17152 16874 16692 16692
query12 188 123 129 123
query13 1247 446 330 330
query14 6296 3244 2977 2977
query14_1 2840 2816 2887 2816
query15 204 188 176 176
query16 1000 462 441 441
query17 1061 681 574 574
query18 2548 430 352 352
query19 205 199 176 176
query20 138 130 128 128
query21 218 137 124 124
query22 4980 5802 5371 5371
query23 17550 17215 16965 16965
query23_1 17027 17293 17120 17120
query24 7195 1641 1238 1238
query24_1 1234 1238 1228 1228
query25 564 484 444 444
query26 1230 262 162 162
query27 2766 474 296 296
query28 4516 1838 1850 1838
query29 817 575 500 500
query30 311 246 224 224
query31 887 722 628 628
query32 84 75 74 74
query33 528 341 285 285
query34 935 908 582 582
query35 665 664 589 589
query36 1087 1147 978 978
query37 134 96 91 91
query38 2925 2913 2908 2908
query39 896 865 839 839
query39_1 826 822 834 822
query40 239 156 141 141
query41 67 63 63 63
query42 108 107 105 105
query43 396 386 363 363
query44
query45 207 190 182 182
query46 880 969 629 629
query47 2141 2148 2031 2031
query48 311 327 243 243
query49 641 474 397 397
query50 686 277 219 219
query51 4030 4191 4087 4087
query52 109 108 96 96
query53 295 344 295 295
query54 320 288 272 272
query55 91 87 86 86
query56 323 327 314 314
query57 1366 1317 1281 1281
query58 300 279 284 279
query59 2574 2640 2503 2503
query60 343 349 344 344
query61 216 143 145 143
query62 626 589 554 554
query63 320 270 280 270
query64 4833 1266 959 959
query65
query66 1433 464 360 360
query67 16607 16417 16309 16309
query68
query69 415 308 274 274
query70 997 989 988 988
query71 340 301 291 291
query72 2808 2680 2378 2378
query73 533 558 320 320
query74 10009 9895 9841 9841
query75 2873 2775 2449 2449
query76 2304 1040 676 676
query77 355 357 304 304
query78 11189 11416 10679 10679
query79 2510 806 619 619
query80 1677 604 513 513
query81 551 275 253 253
query82 759 146 115 115
query83 352 260 240 240
query84 251 127 101 101
query85 895 461 426 426
query86 408 306 297 297
query87 3100 3124 3029 3029
query88 3598 2675 2658 2658
query89 424 369 335 335
query90 2034 175 177 175
query91 157 159 133 133
query92 72 68 74 68
query93 1085 849 510 510
query94 633 312 293 293
query95 592 339 318 318
query96 634 514 228 228
query97 2443 2498 2394 2394
query98 229 217 212 212
query99 1000 994 915 915
Total cold run time: 255599 ms
Total hot run time: 184647 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3961095459
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) Fix tablets permanently invisible to compaction scheduler due to race condition in `CloudTabletMgr::get_tablet` [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #60832:
URL: https://github.com/apache/doris/pull/60832#issuecomment-3961095710
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org