You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "liaoxin01 (via GitHub)" <gi...@apache.org> on 2026/03/24 16:00:59 UTC
[PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
liaoxin01 opened a new pull request, #61684:
URL: https://github.com/apache/doris/pull/61684
## Problem
When a queued memtable flush task starts after `FlushToken::cancel()` has already marked the token shutdown, `_flush_memtable()` can return before the old `flush_running_count++` path but still run the deferred `flush_running_count--`. This can drive the counter below zero and make `cancel()` wait forever for `flush_running_count == 0`.
## Fix
Move `flush_running_count++` to the top of `_flush_memtable()` before registering the deferred cleanup so the running-count accounting stays symmetric on every exit path.
## Validation
- Reasoned from production stack and gdb evidence showing `flush_running_count = -1` while cancel was blocked in `_wait_running_task_finish()`
- Not run locally: the BE UT environment in this workspace would require a full initial `ut_build_ASAN` build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4119764939
<details>
<summary>TPC-H: <b>Total hot run time: 26367 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe5cc860428c9de15e33c5c75517f3c61a19c435, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17636 4512 4281 4281
q2 q3 10649 786 510 510
q4 4676 363 253 253
q5 7558 1191 1015 1015
q6 185 180 149 149
q7 770 862 661 661
q8 9384 1468 1299 1299
q9 5062 4739 4702 4702
q10 6328 1893 1665 1665
q11 462 258 248 248
q12 751 588 468 468
q13 18048 2707 1904 1904
q14 236 231 203 203
q15 q16 732 746 655 655
q17 743 861 400 400
q18 5870 5441 5213 5213
q19 1217 986 613 613
q20 539 496 375 375
q21 4500 1894 1429 1429
q22 348 324 442 324
Total cold run time: 95694 ms
Total hot run time: 26367 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4695 4823 4760 4760
q2 q3 3881 4288 3861 3861
q4 898 1203 805 805
q5 4074 4390 4301 4301
q6 184 176 145 145
q7 1799 1639 1541 1541
q8 2519 2755 2618 2618
q9 7616 7434 7490 7434
q10 3777 3978 3648 3648
q11 507 444 426 426
q12 501 599 439 439
q13 2437 3022 2050 2050
q14 422 385 287 287
q15 q16 743 788 722 722
q17 1247 1378 1362 1362
q18 7283 6991 6717 6717
q19 885 858 917 858
q20 2051 2141 2067 2067
q21 3964 3468 3399 3399
q22 474 460 379 379
Total cold run time: 49957 ms
Total hot run time: 47819 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4120106747
# BE UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/fe5cc860428c9de15e33c5c75517f3c61a19c435_fe5cc860428c9de15e33c5c75517f3c61a19c435/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/fe5cc860428c9de15e33c5c75517f3c61a19c435_fe5cc860428c9de15e33c5c75517f3c61a19c435/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.82% (19870/37615) |
| Line Coverage | 36.31% (185638/511189) |
| Region Coverage | 32.58% (143776/441363) |
| Branch Coverage | 33.79% (62953/186329) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #61684:
URL: https://github.com/apache/doris/pull/61684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4119457197
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4139827935
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #61684:
URL: https://github.com/apache/doris/pull/61684#discussion_r2982709099
##########
be/src/load/memtable/memtable_flush_executor.cpp:
##########
@@ -222,6 +222,7 @@ void FlushToken::_flush_memtable(std::shared_ptr<MemTable> memtable_ptr, int32_t
int64_t submit_task_time) {
signal::set_signal_task_id(_rowset_writer->load_id());
signal::tablet_id = memtable_ptr->tablet_id();
+ _stats.flush_running_count++;
Review Comment:
`flush_running_count` is incremented without holding `_mutex`, while the deferred decrement is performed under `_mutex`. This creates a data-race risk and can still lead to inconsistent observations (including transient negative values) under concurrent reads/writes. Consider protecting the increment with the same `_mutex` (or making `flush_running_count` an atomic and using consistent memory ordering) so the counter updates are thread-safe and consistently synchronized.
```suggestion
{
std::lock_guard<std::mutex> lock(_mutex);
_stats.flush_running_count++;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4120651084
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/61684_fe5cc860428c9de15e33c5c75517f3c61a19c435_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/61684_fe5cc860428c9de15e33c5c75517f3c61a19c435_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.62% (26378/36832) |
| Line Coverage | 54.44% (277424/509637) |
| Region Coverage | 51.64% (230063/445503) |
| Branch Coverage | 53.10% (99234/186897) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "liaoxin01 (via GitHub)" <gi...@apache.org>.
liaoxin01 commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4119549634
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4119833777
<details>
<summary>TPC-DS: <b>Total hot run time: 169088 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe5cc860428c9de15e33c5c75517f3c61a19c435, data reload: false
query5 4326 639 525 525
query6 346 230 197 197
query7 4225 481 259 259
query8 343 249 228 228
query9 8693 2691 2718 2691
query10 496 391 364 364
query11 6991 5081 4876 4876
query12 189 128 126 126
query13 1274 459 354 354
query14 5807 3755 3481 3481
query14_1 2863 2870 2814 2814
query15 214 196 177 177
query16 983 467 466 466
query17 902 732 622 622
query18 2453 451 359 359
query19 217 214 190 190
query20 150 132 126 126
query21 216 138 113 113
query22 13218 14054 14619 14054
query23 16770 16300 15902 15902
query23_1 16307 15825 15828 15825
query24 7218 1602 1221 1221
query24_1 1241 1238 1233 1233
query25 560 508 403 403
query26 1240 259 149 149
query27 2785 473 289 289
query28 4502 1832 1819 1819
query29 832 570 470 470
query30 303 236 190 190
query31 993 929 872 872
query32 79 74 68 68
query33 513 332 300 300
query34 891 873 527 527
query35 646 685 597 597
query36 1078 1144 969 969
query37 132 96 85 85
query38 2972 2963 2962 2962
query39 868 834 832 832
query39_1 792 803 809 803
query40 238 156 134 134
query41 64 58 57 57
query42 256 253 254 253
query43 242 290 216 216
query44
query45 201 191 176 176
query46 877 976 610 610
query47 2141 2121 2062 2062
query48 327 316 230 230
query49 631 465 367 367
query50 680 270 206 206
query51 4037 4082 4061 4061
query52 262 269 253 253
query53 289 332 291 291
query54 307 276 257 257
query55 91 87 86 86
query56 311 320 333 320
query57 1918 1694 1707 1694
query58 282 292 268 268
query59 2796 2961 2754 2754
query60 341 334 319 319
query61 158 155 158 155
query62 615 588 543 543
query63 317 273 280 273
query64 5043 1267 974 974
query65
query66 1465 462 349 349
query67 24285 24418 24262 24262
query68
query69 425 321 283 283
query70 972 981 949 949
query71 348 310 311 310
query72 2832 2653 2456 2456
query73 544 546 324 324
query74 9620 9576 9429 9429
query75 2852 2790 2460 2460
query76 2302 1033 674 674
query77 368 388 314 314
query78 10926 11047 10475 10475
query79 2453 775 579 579
query80 1750 628 568 568
query81 547 258 235 235
query82 1037 153 120 120
query83 344 259 248 248
query84 305 123 93 93
query85 901 500 434 434
query86 417 339 296 296
query87 3163 3121 2973 2973
query88 3544 2653 2648 2648
query89 429 370 372 370
query90 2016 177 174 174
query91 167 166 134 134
query92 75 75 69 69
query93 1026 870 505 505
query94 672 319 284 284
query95 581 346 377 346
query96 664 508 231 231
query97 2450 2457 2422 2422
query98 236 225 218 218
query99 1006 1012 909 909
Total cold run time: 252262 ms
Total hot run time: 169088 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) keep FlushToken running count symmetric on cancel [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61684:
URL: https://github.com/apache/doris/pull/61684#issuecomment-4139822237
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org