You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sollhui (via GitHub)" <gi...@apache.org> on 2025/10/14 07:44:12 UTC
[PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
sollhui opened a new pull request, #56948:
URL: https://github.com/apache/doris/pull/56948
### What problem does this PR solve?
In some cases, the size of `segments` may be relatively small, which not only affects read performance but may also cause transaction commit failures in cloud mode for fdb value limit.
We introduce the adaptive memtable write buffer size scheme, which adaptively increases the write buffer size when there is enough memory to avoid the generation of small segments as much as possible.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405552691
<details>
<summary>ClickBench: <b>Total hot run time: 29.9 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 697085113e3c0fa673b1f65c0c05c126551c7f66, data reload: false
query1 0.05 0.05 0.05
query2 0.10 0.05 0.06
query3 0.25 0.08 0.08
query4 1.61 0.12 0.12
query5 0.27 0.28 0.24
query6 1.19 0.64 0.64
query7 0.02 0.03 0.03
query8 0.05 0.04 0.04
query9 0.62 0.53 0.53
query10 0.58 0.60 0.58
query11 0.17 0.11 0.11
query12 0.16 0.12 0.12
query13 0.64 0.63 0.62
query14 1.02 1.04 1.04
query15 0.90 0.86 0.84
query16 0.40 0.40 0.43
query17 1.05 1.08 1.05
query18 0.22 0.20 0.20
query19 1.93 1.80 1.85
query20 0.02 0.01 0.02
query21 15.47 0.92 0.59
query22 0.76 1.22 0.62
query23 14.92 1.39 0.62
query24 7.00 1.20 0.36
query25 0.32 0.19 0.11
query26 0.58 0.16 0.13
query27 0.08 0.06 0.06
query28 9.32 1.37 0.92
query29 12.58 3.89 3.24
query30 0.29 0.15 0.11
query31 2.84 0.59 0.39
query32 3.26 0.58 0.48
query33 3.12 3.10 3.09
query34 16.26 5.50 4.86
query35 5.00 4.87 4.93
query36 0.71 0.52 0.52
query37 0.10 0.08 0.07
query38 0.06 0.05 0.04
query39 0.03 0.03 0.03
query40 0.19 0.16 0.14
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 104.31 s
Total hot run time: 29.9 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401391595
<details>
<summary>ClickBench: <b>Total hot run time: 30.63 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 41affff5c078c617e19c81ad1b586e085ba12fd6, data reload: false
query1 0.05 0.05 0.04
query2 0.10 0.06 0.05
query3 0.25 0.09 0.08
query4 1.60 0.12 0.11
query5 0.27 0.26 0.25
query6 1.18 0.66 0.63
query7 0.03 0.03 0.02
query8 0.06 0.04 0.05
query9 0.62 0.52 0.52
query10 0.57 0.61 0.57
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.64 0.61 0.61
query14 1.00 1.02 1.02
query15 0.86 0.85 0.86
query16 0.39 0.40 0.39
query17 1.04 1.05 1.06
query18 0.22 0.20 0.20
query19 1.98 1.90 1.82
query20 0.01 0.01 0.01
query21 15.46 0.93 0.58
query22 0.76 1.07 0.63
query23 15.08 1.38 0.69
query24 6.74 1.52 1.08
query25 0.48 0.11 0.15
query26 0.63 0.17 0.13
query27 0.06 0.06 0.05
query28 9.90 1.34 0.95
query29 12.60 4.01 3.30
query30 0.28 0.15 0.11
query31 2.83 0.60 0.38
query32 3.25 0.57 0.47
query33 3.10 3.07 3.14
query34 16.15 5.47 4.83
query35 4.88 4.93 4.88
query36 0.69 0.52 0.50
query37 0.10 0.08 0.07
query38 0.08 0.05 0.04
query39 0.03 0.03 0.03
query40 0.18 0.16 0.15
query41 0.08 0.04 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.05
Total cold run time: 104.63 s
Total hot run time: 30.63 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405531581
<details>
<summary>TPC-DS: <b>Total hot run time: 188340 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 697085113e3c0fa673b1f65c0c05c126551c7f66, data reload: false
query1 1075 422 411 411
query2 6135 1724 1739 1724
query3 4262 222 226 222
query4 25767 23463 23029 23029
query5 4546 640 486 486
query6 314 237 228 228
query7 4326 502 306 306
query8 307 259 273 259
query9 6778 2595 2561 2561
query10 508 334 302 302
query11 15537 15282 14762 14762
query12 195 115 112 112
query13 1649 549 428 428
query14 11233 9318 9333 9318
query15 216 204 201 201
query16 7692 701 531 531
query17 1606 798 676 676
query18 2022 426 319 319
query19 215 205 168 168
query20 138 126 119 119
query21 217 145 125 125
query22 4113 4134 4127 4127
query23 34047 33190 32989 32989
query24 7988 2391 2385 2385
query25 582 499 444 444
query26 1239 271 160 160
query27 2729 499 355 355
query28 4416 2163 2179 2163
query29 809 613 478 478
query30 309 232 214 214
query31 911 822 756 756
query32 86 88 75 75
query33 589 378 326 326
query34 855 859 531 531
query35 803 891 765 765
query36 983 999 929 929
query37 123 108 95 95
query38 3566 3502 3454 3454
query39 1478 1461 1482 1461
query40 221 129 123 123
query41 65 62 63 62
query42 128 116 110 110
query43 500 509 483 483
query44 1386 850 814 814
query45 195 185 179 179
query46 883 1004 637 637
query47 1786 1791 1709 1709
query48 401 420 326 326
query49 774 532 417 417
query50 668 696 414 414
query51 3918 3857 3860 3857
query52 120 111 99 99
query53 251 283 207 207
query54 599 597 548 548
query55 94 90 88 88
query56 345 340 324 324
query57 1182 1230 1110 1110
query58 289 303 294 294
query59 2646 2684 2519 2519
query60 356 347 330 330
query61 160 150 165 150
query62 789 733 677 677
query63 239 199 205 199
query64 4519 1157 828 828
query65 4054 3958 3944 3944
query66 1081 517 357 357
query67 15407 15189 14994 14994
query68 8485 949 595 595
query69 533 330 304 304
query70 1412 1292 1282 1282
query71 460 346 322 322
query72 5755 4957 4884 4884
query73 600 568 362 362
query74 8862 8823 8880 8823
query75 3355 3334 2812 2812
query76 3571 1168 765 765
query77 483 408 344 344
query78 9743 9626 8835 8835
query79 2608 838 595 595
query80 631 565 500 500
query81 502 279 256 256
query82 254 236 149 149
query83 265 269 244 244
query84 257 113 96 96
query85 885 469 429 429
query86 379 304 323 304
query87 3704 3756 3641 3641
query88 3488 2296 2292 2292
query89 399 332 308 308
query90 1990 230 227 227
query91 244 167 140 140
query92 88 76 69 69
query93 1970 1006 655 655
query94 765 445 339 339
query95 404 328 312 312
query96 499 588 289 289
query97 2960 2991 2869 2869
query98 246 207 213 207
query99 1339 1422 1298 1298
Total cold run time: 270556 ms
Total hot run time: 188340 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401653979
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3400541307
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401779126
# BE UT Coverage Report
Increment line coverage `66.67% (10/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/41affff5c078c617e19c81ad1b586e085ba12fd6_41affff5c078c617e19c81ad1b586e085ba12fd6/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/41affff5c078c617e19c81ad1b586e085ba12fd6_41affff5c078c617e19c81ad1b586e085ba12fd6/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.55% (17755/33790) |
| Line Coverage | 37.75% (161468/427761) |
| Region Coverage | 32.20% (123179/382593) |
| Branch Coverage | 33.63% (54121/160935) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #56948:
URL: https://github.com/apache/doris/pull/56948
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401043994
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3406215926
# BE Regression && UT Coverage Report
Increment line coverage `70.97% (22/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56948_697085113e3c0fa673b1f65c0c05c126551c7f66_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56948_697085113e3c0fa673b1f65c0c05c126551c7f66_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.20% (23615/33169) |
| Line Coverage | 57.62% (246477/427792) |
| Region Coverage | 52.69% (204265/387666) |
| Branch Coverage | 54.53% (88281/161904) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401045254
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3408842754
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405104363
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401375037
<details>
<summary>TPC-DS: <b>Total hot run time: 190045 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 41affff5c078c617e19c81ad1b586e085ba12fd6, data reload: false
query1 1095 436 416 416
query2 6571 1704 1691 1691
query3 6761 221 218 218
query4 26886 23745 23090 23090
query5 5828 644 498 498
query6 342 255 221 221
query7 4652 510 296 296
query8 323 284 274 274
query9 8708 2560 2547 2547
query10 583 326 297 297
query11 15342 15172 14973 14973
query12 184 128 118 118
query13 1718 560 458 458
query14 12635 9195 9201 9195
query15 254 192 175 175
query16 7817 677 500 500
query17 1571 788 705 705
query18 2125 481 366 366
query19 302 226 176 176
query20 145 136 138 136
query21 237 148 118 118
query22 4804 4919 4576 4576
query23 34661 33825 34070 33825
query24 8359 2466 2527 2466
query25 620 568 511 511
query26 1734 285 163 163
query27 2829 527 373 373
query28 4392 2245 2216 2216
query29 800 653 549 549
query30 356 237 210 210
query31 897 922 774 774
query32 89 74 68 68
query33 592 391 347 347
query34 834 872 524 524
query35 832 858 795 795
query36 1013 1035 935 935
query37 121 110 95 95
query38 3729 3776 3718 3718
query39 1565 1467 1496 1467
query40 226 135 139 135
query41 73 61 61 61
query42 119 108 111 108
query43 487 496 472 472
query44 1314 830 829 829
query45 193 187 176 176
query46 844 1002 645 645
query47 1812 1890 1731 1731
query48 379 410 310 310
query49 743 493 407 407
query50 638 683 407 407
query51 3915 3955 3869 3869
query52 113 105 105 105
query53 226 260 202 202
query54 600 579 512 512
query55 91 83 85 83
query56 310 331 320 320
query57 1176 1201 1142 1142
query58 286 276 272 272
query59 2511 2591 2515 2515
query60 342 337 332 332
query61 152 149 200 149
query62 812 712 656 656
query63 229 195 185 185
query64 4409 1169 847 847
query65 4036 3945 3951 3945
query66 1037 415 329 329
query67 15546 15509 15176 15176
query68 6391 948 598 598
query69 506 323 285 285
query70 1371 1291 1311 1291
query71 497 347 305 305
query72 5850 4929 4869 4869
query73 526 591 354 354
query74 9425 8937 8498 8498
query75 3757 3414 2813 2813
query76 3209 1158 735 735
query77 796 398 326 326
query78 9518 9792 8942 8942
query79 2747 799 606 606
query80 668 571 469 469
query81 523 271 229 229
query82 224 161 139 139
query83 280 260 242 242
query84 256 109 102 102
query85 875 481 405 405
query86 341 305 284 284
query87 3780 3763 3653 3653
query88 4104 2209 2206 2206
query89 377 324 293 293
query90 2038 211 215 211
query91 158 169 133 133
query92 88 67 65 65
query93 2323 971 643 643
query94 639 438 349 349
query95 396 321 309 309
query96 489 573 279 279
query97 2924 2984 2867 2867
query98 251 214 214 214
query99 1351 1401 1331 1331
Total cold run time: 281603 ms
Total hot run time: 190045 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3402140119
# BE Regression && UT Coverage Report
Increment line coverage `66.67% (10/15)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/56948_41affff5c078c617e19c81ad1b586e085ba12fd6_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/56948_41affff5c078c617e19c81ad1b586e085ba12fd6_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.14% (23565/33125) |
| Line Coverage | 57.56% (246066/427507) |
| Region Coverage | 52.52% (203549/387543) |
| Branch Coverage | 54.46% (88135/161824) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3400517168
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3400774541
<details>
<summary>ClickBench: <b>Total hot run time: 30.06 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18b7b6acd9549fdf87e638ad2d7f119c2245527b, data reload: false
query1 0.05 0.05 0.05
query2 0.10 0.05 0.06
query3 0.25 0.09 0.09
query4 1.63 0.13 0.13
query5 0.29 0.28 0.26
query6 1.27 0.68 0.64
query7 0.03 0.03 0.03
query8 0.06 0.05 0.05
query9 0.64 0.54 0.52
query10 0.60 0.58 0.58
query11 0.19 0.11 0.12
query12 0.15 0.12 0.12
query13 0.63 0.62 0.62
query14 1.01 1.01 1.04
query15 0.87 0.89 0.87
query16 0.40 0.40 0.39
query17 1.03 1.09 1.04
query18 0.21 0.20 0.20
query19 1.91 1.89 1.87
query20 0.02 0.01 0.01
query21 15.42 0.89 0.57
query22 0.77 1.20 0.61
query23 15.01 1.43 0.68
query24 6.85 1.39 0.38
query25 0.53 0.26 0.12
query26 0.62 0.16 0.14
query27 0.07 0.05 0.06
query28 8.87 1.39 0.92
query29 12.60 3.89 3.22
query30 0.28 0.14 0.11
query31 2.82 0.63 0.38
query32 3.24 0.55 0.48
query33 3.14 3.08 3.13
query34 16.03 5.42 4.84
query35 4.92 4.95 4.93
query36 0.68 0.52 0.51
query37 0.11 0.07 0.08
query38 0.07 0.05 0.05
query39 0.04 0.03 0.03
query40 0.18 0.16 0.15
query41 0.09 0.04 0.03
query42 0.04 0.03 0.03
query43 0.04 0.04 0.04
Total cold run time: 103.76 s
Total hot run time: 30.06 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3401654186
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #56948:
URL: https://github.com/apache/doris/pull/56948#discussion_r2428309679
##########
be/src/olap/memtable.h:
##########
@@ -213,6 +214,21 @@ class MemTable {
// Used to wrapped by to_block to do exception handle logic
Status _to_block(std::unique_ptr<vectorized::Block>* res);
+ int64_t _adaptive_write_buffer_size() const {
+ const int64_t current_load_mem_value = MemoryProfile::load_current_usage();
+ int64_t factor = 4;
+ // Memory usage intervals:
+ // (80 %, 100 %] → 1× buffer
+ // (50 %, 80 %] → 2× buffer
+ // [0 %, 50 %] → 4× buffer
+ if (current_load_mem_value > (_load_mem_limit * 4) / 5) { // > 80 %
+ factor = 1;
+ } else if (current_load_mem_value > _load_mem_limit / 2) { // > 50 %
+ factor = 2;
+ }
+ return config::write_buffer_size * factor;
+ }
Review Comment:
put it to .cpp
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](memtable) support adaptive memtable write buffer size [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #56948:
URL: https://github.com/apache/doris/pull/56948#issuecomment-3405980751
# BE UT Coverage Report
Increment line coverage `48.39% (15/31)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/697085113e3c0fa673b1f65c0c05c126551c7f66_697085113e3c0fa673b1f65c0c05c126551c7f66/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/697085113e3c0fa673b1f65c0c05c126551c7f66_697085113e3c0fa673b1f65c0c05c126551c7f66/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.57% (17789/33837) |
| Line Coverage | 37.77% (161665/428049) |
| Region Coverage | 32.20% (123249/382719) |
| Branch Coverage | 33.63% (54149/161015) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org