You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "eldenmoon (via GitHub)" <gi...@apache.org> on 2026/03/31 11:30:29 UTC
[PR] [fix](variant): reduce doc compaction writer peak memory [doris]
eldenmoon opened a new pull request, #61955:
URL: https://github.com/apache/doris/pull/61955
- cap doc-value path reserve and reuse exact unique-path counts when available
- release doc write plans before doc-value writing to avoid overlapping peaks
- eagerly finish and flush compact-writer subcolumns, and make finish/write_data idempotent
- avoid merge/sort when bucket = 1
- add unit coverage for expected_unique_paths and compact-writer roundtrip/idempotence
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4166933793
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4167095780
skip check_coverage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant): reduce doc compaction writer peak memory [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4161958744
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4162379572
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon merged PR #61955:
URL: https://github.com/apache/doris/pull/61955
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4162193935
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4162534311
<details>
<summary>TPC-H: <b>Total hot run time: 26526 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eab7f32594deb3d7d9fd6deab580061312710b9d, data reload: false
------ Round 1 ----------------------------------
orders Doris NULL NULL 0 0 0 NULL 0 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 17624 4534 4263 4263
q2 q3 10647 769 528 528
q4 4678 355 245 245
q5 7563 1211 1019 1019
q6 175 171 146 146
q7 784 848 668 668
q8 9308 1446 1342 1342
q9 4961 4756 4723 4723
q10 6322 1930 1670 1670
q11 483 257 240 240
q12 749 576 465 465
q13 18050 2681 1916 1916
q14 226 230 208 208
q15 q16 748 739 665 665
q17 733 855 434 434
q18 5940 5420 5208 5208
q19 1114 974 662 662
q20 537 487 374 374
q21 4665 1815 1418 1418
q22 335 332 435 332
Total cold run time: 95642 ms
Total hot run time: 26526 ms
----- Round 2, with runtime_filter_mode=off -----
orders Doris NULL NULL 150000000 42 6422171781 NULL 22778155 NULL NULL 2023-12-26 18:27:23 2023-12-26 18:42:55 NULL utf-8 NULL NULL
============================================
q1 4781 4824 4599 4599
q2 q3 3917 4369 3846 3846
q4 856 1185 795 795
q5 4069 4375 4326 4326
q6 193 172 145 145
q7 1835 1684 1580 1580
q8 2507 2783 2971 2783
q9 7578 7294 7320 7294
q10 3766 4088 3684 3684
q11 504 449 440 440
q12 530 626 450 450
q13 2483 2984 2105 2105
q14 273 301 283 283
q15 q16 720 744 719 719
q17 1182 1369 1333 1333
q18 7001 6835 6716 6716
q19 1059 959 894 894
q20 2083 2154 1975 1975
q21 3978 3464 3376 3376
q22 461 438 394 394
Total cold run time: 49776 ms
Total hot run time: 47737 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4166933920
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4162602651
<details>
<summary>TPC-DS: <b>Total hot run time: 169678 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eab7f32594deb3d7d9fd6deab580061312710b9d, data reload: false
query5 4328 649 515 515
query6 333 226 202 202
query7 4219 481 263 263
query8 341 245 229 229
query9 8742 2658 2672 2658
query10 485 376 345 345
query11 7047 5087 4864 4864
query12 186 137 124 124
query13 1278 467 344 344
query14 5806 3659 3511 3511
query14_1 2864 2822 2798 2798
query15 205 196 175 175
query16 981 504 452 452
query17 894 762 594 594
query18 2431 441 338 338
query19 210 208 183 183
query20 127 123 125 123
query21 214 132 114 114
query22 13350 15219 14617 14617
query23 16541 16271 16077 16077
query23_1 15706 15673 15760 15673
query24 7241 1630 1218 1218
query24_1 1233 1220 1226 1220
query25 564 502 405 405
query26 1234 267 153 153
query27 2784 474 296 296
query28 4505 1855 1807 1807
query29 836 566 487 487
query30 304 231 192 192
query31 1012 950 878 878
query32 83 71 69 69
query33 518 329 291 291
query34 890 859 515 515
query35 646 690 605 605
query36 1097 1122 922 922
query37 146 98 86 86
query38 3001 2943 2921 2921
query39 853 828 801 801
query39_1 799 799 808 799
query40 230 152 136 136
query41 62 59 58 58
query42 263 257 263 257
query43 248 248 223 223
query44
query45 196 186 187 186
query46 873 1022 610 610
query47 2856 2144 2043 2043
query48 316 339 225 225
query49 645 487 391 391
query50 704 278 215 215
query51 4011 4059 4064 4059
query52 265 266 250 250
query53 296 349 290 290
query54 294 274 271 271
query55 97 87 83 83
query56 330 331 311 311
query57 1914 1735 1860 1735
query58 283 280 269 269
query59 2798 2939 2730 2730
query60 343 345 339 339
query61 162 158 158 158
query62 633 592 545 545
query63 309 278 275 275
query64 5113 1305 1021 1021
query65
query66 1468 465 353 353
query67 24281 24339 24223 24223
query68
query69 409 317 297 297
query70 917 997 962 962
query71 339 318 303 303
query72 3070 2967 2523 2523
query73 542 555 321 321
query74 9618 9543 9455 9455
query75 2873 2781 2475 2475
query76 2290 1044 679 679
query77 404 392 308 308
query78 10877 11051 10495 10495
query79 2389 753 586 586
query80 1785 625 557 557
query81 552 257 225 225
query82 1004 150 119 119
query83 346 263 242 242
query84 301 120 105 105
query85 927 524 469 469
query86 404 302 298 298
query87 3180 3107 3014 3014
query88 3542 2672 2649 2649
query89 438 372 340 340
query90 2017 182 181 181
query91 172 167 137 137
query92 85 73 68 68
query93 1019 851 517 517
query94 654 314 289 289
query95 606 409 317 317
query96 647 516 224 224
query97 2461 2467 2391 2391
query98 231 221 246 221
query99 1012 994 919 919
Total cold run time: 252137 ms
Total hot run time: 169678 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](variant) reduce doc compaction writer peak memory [doris]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on PR #61955:
URL: https://github.com/apache/doris/pull/61955#issuecomment-4162018302
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org