You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2025/12/26 03:06:52 UTC
[PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Mryange opened a new pull request, #59374:
URL: https://github.com/apache/doris/pull/59374
### What problem does this PR solve?
Previously, when increasing memory usage the code did not account for the channels' memory, but it did include it when freeing memory.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3692052580
# BE UT Coverage Report
Increment line coverage `100.00% (12/12)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/d993a41e4d5a2d3d829b99339131d1f14f675092_d993a41e4d5a2d3d829b99339131d1f14f675092/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/d993a41e4d5a2d3d829b99339131d1f14f675092_d993a41e4d5a2d3d829b99339131d1f14f675092/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 53.40% (18952/35489) |
| Line Coverage | 39.27% (175803/447680) |
| Region Coverage | 33.85% (136069/402007) |
| Branch Coverage | 34.76% (58731/168940) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3695261962
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3692334369
<details>
<summary>TPC-DS: <b>Total hot run time: 180172 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d993a41e4d5a2d3d829b99339131d1f14f675092, data reload: false
query5 5011 598 454 454
query6 338 253 233 233
query7 4235 470 285 285
query8 309 257 260 257
query9 8799 2525 2521 2521
query10 546 381 335 335
query11 15743 15315 14844 14844
query12 189 122 116 116
query13 1264 500 393 393
query14 6421 3000 2788 2788
query14_1 2692 2655 2679 2655
query15 210 194 178 178
query16 912 484 441 441
query17 1134 712 617 617
query18 2687 443 354 354
query19 236 235 211 211
query20 125 117 114 114
query21 224 139 116 116
query22 4132 4179 4074 4074
query23 16664 16257 16002 16002
query23_1 16121 16099 16155 16099
query24 7342 1669 1249 1249
query24_1 1249 1213 1264 1213
query25 585 510 445 445
query26 1254 277 171 171
query27 2732 459 311 311
query28 4472 2113 2098 2098
query29 848 591 504 504
query30 316 233 220 220
query31 814 721 618 618
query32 87 69 78 69
query33 546 349 324 324
query34 903 915 537 537
query35 753 806 705 705
query36 880 922 827 827
query37 126 93 77 77
query38 2981 3081 2986 2986
query39 750 769 716 716
query39_1 697 716 699 699
query40 230 135 118 118
query41 65 63 62 62
query42 105 104 106 104
query43 429 445 406 406
query44 1341 746 740 740
query45 187 188 180 180
query46 873 961 616 616
query47 1694 1719 1619 1619
query48 312 331 243 243
query49 638 428 393 393
query50 668 299 220 220
query51 3861 3795 3839 3795
query52 105 114 103 103
query53 316 344 286 286
query54 283 301 255 255
query55 79 76 70 70
query56 305 304 285 285
query57 1194 1171 1082 1082
query58 279 256 254 254
query59 2388 2604 2354 2354
query60 309 308 285 285
query61 167 156 159 156
query62 761 686 663 663
query63 329 293 299 293
query64 4979 1296 988 988
query65 3998 3952 3964 3952
query66 1400 437 314 314
query67 15284 15164 14955 14955
query68 6916 1001 719 719
query69 507 345 310 310
query70 1082 991 976 976
query71 365 303 286 286
query72 6065 5062 5113 5062
query73 715 649 307 307
query74 8968 8904 8678 8678
query75 3206 3154 2853 2853
query76 3898 1146 750 750
query77 535 387 295 295
query78 9518 9712 8863 8863
query79 1261 867 621 621
query80 1439 678 573 573
query81 545 266 233 233
query82 400 134 104 104
query83 273 259 241 241
query84 259 124 99 99
query85 911 497 457 457
query86 333 285 276 276
query87 3222 3182 3207 3182
query88 3272 2310 2307 2307
query89 484 434 401 401
query90 1934 169 156 156
query91 175 172 145 145
query92 71 69 74 69
query93 1109 899 553 553
query94 557 306 266 266
query95 567 389 302 302
query96 600 459 211 211
query97 2298 2279 2253 2253
query98 212 193 193 193
query99 1321 1338 1315 1315
Total cold run time: 260866 ms
Total hot run time: 180172 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3692341251
<details>
<summary>ClickBench: <b>Total hot run time: 27.23 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d993a41e4d5a2d3d829b99339131d1f14f675092, data reload: false
query1 0.08 0.05 0.05
query2 0.10 0.05 0.05
query3 0.26 0.09 0.08
query4 1.60 0.11 0.11
query5 0.27 0.25 0.26
query6 1.19 0.64 0.64
query7 0.03 0.02 0.02
query8 0.05 0.04 0.04
query9 0.56 0.50 0.50
query10 0.55 0.55 0.53
query11 0.15 0.11 0.11
query12 0.16 0.11 0.11
query13 0.61 0.60 0.60
query14 0.99 0.99 0.99
query15 0.82 0.79 0.81
query16 0.39 0.39 0.40
query17 0.98 1.08 1.02
query18 0.22 0.21 0.22
query19 1.93 1.85 1.76
query20 0.02 0.02 0.02
query21 15.42 0.27 0.15
query22 4.79 0.05 0.04
query23 16.06 0.30 0.10
query24 0.96 0.28 0.48
query25 0.08 0.08 0.06
query26 0.15 0.14 0.13
query27 0.07 0.07 0.04
query28 3.79 1.20 1.03
query29 12.63 4.09 3.19
query30 0.28 0.14 0.13
query31 2.82 0.62 0.39
query32 3.24 0.55 0.45
query33 3.03 3.03 3.10
query34 16.93 5.20 4.54
query35 4.57 4.56 4.59
query36 0.65 0.49 0.51
query37 0.11 0.07 0.06
query38 0.07 0.04 0.03
query39 0.05 0.03 0.03
query40 0.18 0.14 0.14
query41 0.09 0.04 0.03
query42 0.04 0.03 0.04
query43 0.05 0.03 0.04
Total cold run time: 97.02 s
Total hot run time: 27.23 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "zclllyybb (via GitHub)" <gi...@apache.org>.
zclllyybb merged PR #59374:
URL: https://github.com/apache/doris/pull/59374
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3695262000
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3692341881
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (12/12)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/59374_d993a41e4d5a2d3d829b99339131d1f14f675092_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/59374_d993a41e4d5a2d3d829b99339131d1f14f675092_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 72.25% (25066/34694) |
| Line Coverage | 58.97% (263272/446488) |
| Region Coverage | 53.77% (218414/406172) |
| Branch Coverage | 55.35% (93816/169504) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3692317811
<details>
<summary>TPC-H: <b>Total hot run time: 36405 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d993a41e4d5a2d3d829b99339131d1f14f675092, data reload: false
------ Round 1 ----------------------------------
q1 17606 4307 4091 4091
q2 2015 336 232 232
q3 10197 1355 782 782
q4 10221 874 318 318
q5 7505 2102 2000 2000
q6 194 170 135 135
q7 997 859 741 741
q8 9362 1432 1208 1208
q9 6930 5309 5357 5309
q10 6882 2395 1984 1984
q11 565 325 299 299
q12 650 740 562 562
q13 17770 3686 3057 3057
q14 291 285 271 271
q15 591 509 506 506
q16 703 681 628 628
q17 687 839 507 507
q18 7514 7610 7882 7610
q19 1235 1028 646 646
q20 423 377 254 254
q21 4632 4228 4364 4228
q22 1146 1111 1037 1037
Total cold run time: 108116 ms
Total hot run time: 36405 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4404 4342 4551 4342
q2 354 424 308 308
q3 2443 2855 2498 2498
q4 1486 1907 1474 1474
q5 4645 4540 4433 4433
q6 211 161 128 128
q7 1982 1941 1811 1811
q8 2721 2479 2574 2479
q9 7614 7051 6895 6895
q10 2906 3088 2677 2677
q11 602 513 485 485
q12 644 728 565 565
q13 3291 3610 3002 3002
q14 267 274 251 251
q15 534 504 485 485
q16 613 664 605 605
q17 1130 1449 1413 1413
q18 7237 7207 7025 7025
q19 889 839 868 839
q20 1881 1952 1787 1787
q21 4570 4350 4156 4156
q22 1104 994 991 991
Total cold run time: 51528 ms
Total hot run time: 48649 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3691982280
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3691983133
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](profile) Fix inaccurate accounting of memory_used_counter in ExchangeSink [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #59374:
URL: https://github.com/apache/doris/pull/59374#issuecomment-3692258684
run performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org