You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2025/04/28 15:26:20 UTC
[PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
xinyiZzz opened a new pull request, #50525:
URL: https://github.com/apache/doris/pull/50525
Prefer use `limiter_mem_tracker`, which is faster than `limiter_mem_tracker_sptr`.
when multiple threads hold the same `std::shared_ptr` at the same time, modifying the `std::shared_ptr` reference count will be expensive when there is high concurrency.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835675483
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835634926
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835633942
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835918245
<details>
<summary>TPC-H: <b>Total hot run time: 40232 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca21f2842b041b83b64853d263435cf5359d2f4d, data reload: false
------ Round 1 ----------------------------------
q1 17600 6797 6623 6623
q2 2077 204 176 176
q3 10530 1078 1173 1078
q4 10533 781 867 781
q5 7747 2859 2888 2859
q6 218 139 137 137
q7 974 619 615 615
q8 9358 1987 2034 1987
q9 6749 6466 6469 6466
q10 7046 2297 2242 2242
q11 457 263 264 263
q12 396 209 208 208
q13 17804 2980 2997 2980
q14 252 207 238 207
q15 510 466 463 463
q16 497 411 383 383
q17 979 534 548 534
q18 7428 6789 6742 6742
q19 1395 1077 1052 1052
q20 470 205 208 205
q21 4021 3238 3234 3234
q22 1122 997 1010 997
Total cold run time: 108163 ms
Total hot run time: 40232 ms
----- Round 2, with runtime_filter_mode=off -----
q1 6605 7065 6582 6582
q2 334 224 228 224
q3 2934 2791 2968 2791
q4 2049 1876 1815 1815
q5 5754 5719 5770 5719
q6 201 129 130 129
q7 2228 1810 1794 1794
q8 3419 3583 3541 3541
q9 8927 8953 8848 8848
q10 3576 3544 3545 3544
q11 593 501 485 485
q12 805 597 638 597
q13 9153 3155 3197 3155
q14 296 267 268 267
q15 519 472 470 470
q16 494 452 438 438
q17 1815 1638 1624 1624
q18 8340 7920 7697 7697
q19 1679 1619 1590 1590
q20 2108 1815 1820 1815
q21 5558 5348 5361 5348
q22 1135 1070 1024 1024
Total cold run time: 68522 ms
Total hot run time: 59497 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #50525:
URL: https://github.com/apache/doris/pull/50525
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835947404
# BE UT Coverage Report
Increment line coverage `80.00% (16/20)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/ca21f2842b041b83b64853d263435cf5359d2f4d_ca21f2842b041b83b64853d263435cf5359d2f4d/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/ca21f2842b041b83b64853d263435cf5359d2f4d_ca21f2842b041b83b64853d263435cf5359d2f4d/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 40.51% (10638/26259) |
| Line Coverage | 31.25% (90140/288471) |
| Region Coverage | 30.38% (46539/153181) |
| Branch Coverage | 26.89% (23812/88568) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835949221
<details>
<summary>TPC-DS: <b>Total hot run time: 197342 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca21f2842b041b83b64853d263435cf5359d2f4d, data reload: false
query1 1290 902 918 902
query2 6274 1891 1923 1891
query3 10912 4536 4369 4369
query4 61845 29612 23437 23437
query5 5233 472 464 464
query6 401 181 203 181
query7 5493 308 318 308
query8 311 226 227 226
query9 8641 2626 2608 2608
query10 461 277 261 261
query11 17456 15405 15820 15405
query12 162 110 102 102
query13 1413 417 420 417
query14 10467 7316 6750 6750
query15 205 179 174 174
query16 7157 511 486 486
query17 1119 600 604 600
query18 1839 328 343 328
query19 213 164 159 159
query20 118 108 110 108
query21 203 103 104 103
query22 4698 4454 4655 4454
query23 34979 34404 34105 34105
query24 6110 2871 2934 2871
query25 549 442 427 427
query26 664 172 175 172
query27 2067 365 350 350
query28 4070 2217 2165 2165
query29 724 478 457 457
query30 245 162 169 162
query31 979 814 825 814
query32 73 59 61 59
query33 420 303 303 303
query34 927 499 512 499
query35 842 731 752 731
query36 1082 978 967 967
query37 109 63 67 63
query38 4119 4014 3974 3974
query39 1527 1509 1444 1444
query40 204 103 107 103
query41 49 49 51 49
query42 115 108 104 104
query43 520 487 479 479
query44 1168 820 829 820
query45 185 168 168 168
query46 1140 736 738 736
query47 1970 1941 1943 1941
query48 481 389 385 385
query49 751 419 427 419
query50 871 443 420 420
query51 7294 7316 7305 7305
query52 108 91 94 91
query53 263 192 191 191
query54 573 499 481 481
query55 86 82 76 76
query56 273 244 254 244
query57 1291 1194 1128 1128
query58 219 239 227 227
query59 3223 2974 3003 2974
query60 276 253 256 253
query61 117 116 130 116
query62 764 670 679 670
query63 222 190 185 185
query64 1396 658 635 635
query65 3246 3249 3215 3215
query66 722 293 317 293
query67 15848 15669 15639 15639
query68 4146 599 575 575
query69 431 259 264 259
query70 1115 1072 1129 1072
query71 347 287 243 243
query72 6351 4134 4201 4134
query73 753 356 364 356
query74 10149 9363 9109 9109
query75 3397 2586 2682 2586
query76 2069 1036 1106 1036
query77 506 278 272 272
query78 10610 9670 9572 9572
query79 1989 607 608 607
query80 1392 434 432 432
query81 514 222 224 222
query82 1247 89 87 87
query83 278 147 147 147
query84 281 82 86 82
query85 1046 315 292 292
query86 385 295 310 295
query87 4447 4278 4277 4277
query88 4070 2394 2356 2356
query89 413 293 289 289
query90 2019 187 187 187
query91 187 150 154 150
query92 67 53 49 49
query93 2925 565 561 561
query94 814 300 295 295
query95 358 261 266 261
query96 619 285 288 285
query97 3314 3127 3165 3127
query98 216 210 198 198
query99 1604 1286 1306 1286
Total cold run time: 316831 ms
Total hot run time: 197342 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] branch-3.0: [fix](memory) Fix `ThreadMemTrackerMgr::limiter_mem_tracker()` performance [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #50525:
URL: https://github.com/apache/doris/pull/50525#issuecomment-2835964836
<details>
<summary>ClickBench: <b>Total hot run time: 31.04 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca21f2842b041b83b64853d263435cf5359d2f4d, data reload: false
query1 0.04 0.03 0.02
query2 0.06 0.03 0.03
query3 0.22 0.06 0.07
query4 1.63 0.11 0.10
query5 0.53 0.50 0.52
query6 1.14 0.72 0.73
query7 0.02 0.02 0.02
query8 0.05 0.03 0.03
query9 0.57 0.48 0.51
query10 0.55 0.54 0.55
query11 0.14 0.10 0.10
query12 0.14 0.12 0.12
query13 0.60 0.61 0.59
query14 0.78 0.79 0.79
query15 0.83 0.83 0.84
query16 0.38 0.41 0.38
query17 1.02 1.00 1.06
query18 0.24 0.22 0.22
query19 1.97 1.79 1.84
query20 0.01 0.02 0.01
query21 15.41 0.60 0.57
query22 2.45 2.29 2.84
query23 16.98 1.05 1.01
query24 3.13 1.10 1.31
query25 0.21 0.17 0.20
query26 0.30 0.13 0.13
query27 0.04 0.04 0.05
query28 10.15 0.49 0.51
query29 12.58 3.20 3.19
query30 0.24 0.06 0.06
query31 2.88 0.39 0.39
query32 3.23 0.47 0.45
query33 2.97 2.98 3.06
query34 17.07 4.50 4.54
query35 4.59 4.60 4.55
query36 0.67 0.48 0.50
query37 0.08 0.06 0.06
query38 0.04 0.04 0.03
query39 0.03 0.02 0.02
query40 0.15 0.13 0.13
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.04 0.03 0.03
Total cold run time: 104.27 s
Total hot run time: 31.04 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org