You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "bobhan1 (via GitHub)" <gi...@apache.org> on 2026/10/09 10:42:02 UTC
[PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
bobhan1 opened a new pull request, #68834:
URL: https://github.com/apache/doris/pull/68834
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: Long-running WAL replay retries can lose their backoff because the linear delay is multiplied using 32-bit integers before the result is converted to `int64_t`. With the default configuration, retry count 1204 makes `(1204 - 10) * 1800 * 1000` exceed `INT32_MAX`, potentially causing replay on every background scan instead of every 30 minutes.
Promote the retry count to `int64_t` before subtraction and multiplication. This preserves the existing backoff formula while keeping the linear delay in 64-bit arithmetic.
### Release note
Fix WAL replay backoff overflow after prolonged retries.
### Check List (For Author)
- Test:
- [x] No need to test or manual test. Explain why:
- [x] Other reason: Minimal arithmetic type promotion. No tests added at the user's request. No automated tests or full build run. No further clang-tidy runs at the user's request; the initial attempt could not analyze the file because the local toolchain could not find `emmintrin.h`.
- Validation: `build-support/clang-format.sh` and `build-support/check-format.sh` passed with clang-format 16.0.6; `git diff --check` passed. Manually checked the default delay at retry counts 1203, 1204, and 1205: 2,152,520,000, 2,154,320,000, and 2,156,120,000 ms, respectively, retaining a 1,800,000 ms increment.
- Behavior changed:
- [x] Yes. Prolonged failed WAL replays retain the configured backoff after the previous 32-bit overflow boundary.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6079280106
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #68834:
URL: https://github.com/apache/doris/pull/68834#discussion_r4229323415
##########
be/src/load/group_commit/wal/wal_table.cpp:
##########
Review Comment:
```suggestion
#ifdef BE_TEST
return true;
#endif
if (config::group_commit_wait_replay_wal_finish) {
return true;
}
int64_t replay_interval = 0;
if (wal_info->get_retry_num() >= config::group_commit_replay_wal_retry_num) {
replay_interval = int64_t(
pow(2, config::group_commit_replay_wal_retry_num) *
config::group_commit_replay_wal_retry_interval_seconds * 1000 +
(int64_t(wal_info->get_retry_num()) - config::group_commit_replay_wal_retry_num) *
config::group_commit_replay_wal_retry_interval_max_seconds * 1000);
} else {
replay_interval = int64_t(pow(2, wal_info->get_retry_num()) *
config::group_commit_replay_wal_retry_interval_seconds * 1000);
}
return UnixMillis() - wal_info->get_start_time_ms() >= replay_interval;
```
##########
be/src/load/group_commit/wal/wal_table.cpp:
##########
Review Comment:
```suggestion
#ifdef BE_TEST
return true;
#endif
if (config::group_commit_wait_replay_wal_finish) {
return true;
}
int64_t replay_interval = 0;
if (wal_info->get_retry_num() >= config::group_commit_replay_wal_retry_num) {
replay_interval = int64_t(
pow(2, config::group_commit_replay_wal_retry_num) *
config::group_commit_replay_wal_retry_interval_seconds * 1000 +
(int64_t(wal_info->get_retry_num()) - config::group_commit_replay_wal_retry_num) *
config::group_commit_replay_wal_retry_interval_max_seconds * 1000);
} else {
replay_interval = int64_t(pow(2, wal_info->get_retry_num()) *
config::group_commit_replay_wal_retry_interval_seconds * 1000);
}
return UnixMillis() - wal_info->get_start_time_ms() >= replay_interval;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6082920476
# BE Regression && UT Coverage Report
Increment line coverage `100% (0/0)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68834_7e4d5108b0ef3a6896ed135d807d15c113d3606f_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68834_7e4d5108b0ef3a6896ed135d807d15c113d3606f_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 76.65% (35189/45909) |
| Line Coverage | 61.86% (397730/643000) |
| Region Coverage | 58.23% (335431/576067) |
| Branch Coverage | 59.15% (154387/261011) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 merged PR #68834:
URL: https://github.com/apache/doris/pull/68834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6079498888
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou commented on code in PR #68834:
URL: https://github.com/apache/doris/pull/68834#discussion_r4229323415
##########
be/src/load/group_commit/wal/wal_table.cpp:
##########
Review Comment:
```suggestion
#ifdef BE_TEST
return true;
#endif
if (config::group_commit_wait_replay_wal_finish) {
return true;
}
int64_t replay_interval = 0;
if (wal_info->get_retry_num() >= config::group_commit_replay_wal_retry_num) {
replay_interval = int64_t(
pow(2, config::group_commit_replay_wal_retry_num) *
config::group_commit_replay_wal_retry_interval_seconds * 1000 +
(int64_t(wal_info->get_retry_num()) - config::group_commit_replay_wal_retry_num) *
config::group_commit_replay_wal_retry_interval_max_seconds * 1000);
} else {
replay_interval = int64_t(pow(2, wal_info->get_retry_num()) *
config::group_commit_replay_wal_retry_interval_seconds * 1000);
}
return UnixMillis() - wal_info->get_start_time_ms() >= replay_interval;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6080976876
<details>
<summary>ClickBench: <b>Total hot run time: 24.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7e4d5108b0ef3a6896ed135d807d15c113d3606f, data reload: false
query1 0.01 0.01 0.01
query2 0.09 0.05 0.05
query3 0.27 0.12 0.11
query4 1.61 0.12 0.13
query5 0.28 0.25 0.26
query6 1.18 0.69 0.69
query7 0.04 0.01 0.01
query8 0.05 0.04 0.04
query9 0.42 0.35 0.36
query10 0.59 0.60 0.57
query11 0.21 0.14 0.16
query12 0.20 0.15 0.15
query13 0.51 0.49 0.51
query14 1.00 0.99 0.97
query15 0.64 0.61 0.61
query16 0.34 0.36 0.32
query17 1.08 1.14 1.13
query18 0.22 0.21 0.20
query19 2.03 1.95 2.04
query20 0.01 0.01 0.02
query21 15.49 0.28 0.14
query22 4.43 0.06 0.05
query23 16.26 0.31 0.13
query24 2.97 0.57 0.58
query25 0.10 0.06 0.05
query26 1.77 0.23 0.16
query27 0.05 0.03 0.04
query28 2.54 0.61 0.40
query29 12.50 4.07 3.25
query30 0.30 0.13 0.13
query31 2.78 0.59 0.34
query32 3.25 0.62 0.52
query33 3.16 3.12 3.18
query34 15.46 4.00 3.37
query35 3.35 3.33 3.36
query36 0.54 0.42 0.43
query37 0.10 0.07 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.03
query40 0.17 0.15 0.15
query41 0.08 0.03 0.03
query42 0.03 0.02 0.02
query43 0.04 0.04 0.03
Total cold run time: 96.23 s
Total hot run time: 24.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6080895568
<details>
<summary>TPC-DS: <b>Total hot run time: 151520 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e4d5108b0ef3a6896ed135d807d15c113d3606f, data reload: false
query5 4307 576 455 455
query6 423 236 185 185
query7 4806 451 224 224
query8 316 181 167 167
query9 8753 3967 3958 3958
query10 457 298 259 259
query11 5971 3455 3190 3190
query12 145 86 84 84
query13 1265 462 316 316
query14 6514 4821 4507 4507
query14_1 4196 4284 4228 4228
query15 211 206 182 182
query16 942 443 384 384
query17 847 652 552 552
query18 2413 425 310 310
query19 190 157 139 139
query20 80 80 80 80
query21 204 131 111 111
query22 13015 13044 12834 12834
query23 13058 12558 12103 12103
query23_1 12352 12302 12230 12230
query24 7102 812 448 448
query24_1 466 460 466 460
query25 528 427 364 364
query26 1284 263 139 139
query27 2766 433 269 269
query28 4626 1922 1926 1922
query29 1558 603 450 450
query30 301 219 180 180
query31 897 775 652 652
query32 137 88 89 88
query33 509 293 247 247
query34 897 849 498 498
query35 736 777 645 645
query36 826 802 740 740
query37 129 103 90 90
query38 1800 1743 1675 1675
query39 727 703 691 691
query39_1 671 660 698 660
query40 218 122 97 97
query41 71 67 68 67
query42 88 86 81 81
query43 368 379 330 330
query44 1296 696 681 681
query45 184 173 166 166
query46 843 965 569 569
query47 2867 3019 2986 2986
query48 289 295 207 207
query49 565 390 285 285
query50 657 268 206 206
query51 10412 10449 10484 10449
query52 79 75 66 66
query53 200 205 148 148
query54 236 181 164 164
query55 67 66 64 64
query56 248 203 207 203
query57 1627 1602 1579 1579
query58 269 255 241 241
query59 2346 2383 2123 2123
query60 270 237 219 219
query61 142 142 142 142
query62 391 339 290 290
query63 197 159 166 159
query64 2754 922 757 757
query65 3423 3381 3356 3356
query66 1768 409 289 289
query67 20068 20087 19994 19994
query68 2999 982 593 593
query69 422 289 245 245
query70 923 834 782 782
query71 297 234 204 204
query72 2672 2576 2174 2174
query73 504 522 291 291
query74 4573 4437 4318 4318
query75 2322 2321 1948 1948
query76 2282 1014 590 590
query77 351 394 296 296
query78 9074 9022 8471 8471
query79 935 881 506 506
query80 624 411 328 328
query81 530 327 274 274
query82 572 132 111 111
query83 251 189 171 171
query84 311 117 106 106
query85 804 418 355 355
query86 344 245 235 235
query87 2033 2007 1805 1805
query88 3544 2687 2672 2672
query89 333 289 257 257
query90 1856 180 164 164
query91 154 143 121 121
query92 95 85 85 85
query93 972 965 565 565
query94 439 318 264 264
query95 580 395 312 312
query96 656 521 229 229
query97 2475 2449 2324 2324
query98 158 148 142 142
query99 777 765 646 646
Total cold run time: 233005 ms
Total hot run time: 151520 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6080733674
<details>
<summary>TPC-H: <b>Total hot run time: 28549 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e4d5108b0ef3a6896ed135d807d15c113d3606f, data reload: false
------ Round 1 ----------------------------------
============================================
q1 17579 5665 5525 5525
q2 2093 337 242 242
q3 10257 1389 845 845
q4 4678 476 337 337
q5 7431 787 528 528
q6 193 187 162 162
q7 753 799 610 610
q8 9240 1368 1006 1006
q9 5651 4448 4407 4407
q10 6799 1287 1021 1021
q11 433 255 236 236
q12 636 398 281 281
q13 18055 3033 2384 2384
q14 276 272 244 244
q15 q16 725 725 668 668
q17 1176 734 555 555
q18 7103 6225 6155 6155
q19 1094 956 608 608
q20 370 332 223 223
q21 4526 2240 2362 2240
q22 358 308 272 272
Total cold run time: 99426 ms
Total hot run time: 28549 ms
----- Round 2, with runtime_filter_mode=off -----
============================================
q1 6121 6075 6075 6075
q2 673 585 572 572
q3 5281 5425 4920 4920
q4 2000 2123 1548 1548
q5 5253 5164 5889 5164
q6 258 193 153 153
q7 2309 2012 1861 1861
q8 3182 2845 2771 2771
q9 8130 8263 8297 8263
q10 3786 3943 3363 3363
q11 652 516 431 431
q12 684 754 530 530
q13 2889 3259 2496 2496
q14 316 316 284 284
q15 q16 702 711 649 649
q17 8355 7513 7383 7383
q18 13129 12330 13147 12330
q19 982 830 814 814
q20 2171 2175 1939 1939
q21 5783 4705 4855 4705
q22 488 448 411 411
Total cold run time: 73144 ms
Total hot run time: 66662 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6092444965
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](be) Fix WAL replay retry backoff overflow [doris]
Posted by "bobhan1 (via GitHub)" <gi...@apache.org>.
bobhan1 commented on PR #68834:
URL: https://github.com/apache/doris/pull/68834#issuecomment-6079306306
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org