You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Yukang-Lian (via GitHub)" <gi...@apache.org> on 2024/07/10 07:58:54 UTC
[PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Yukang-Lian opened a new pull request, #37602:
URL: https://github.com/apache/doris/pull/37602
The default delete bitmap cache is set to 100MB, which can be insufficient and cause performance issues when the amount of user data is large. To mitigate the problem of an inadequate cache, we will take the larger of 5% of the total memory and 100MB as the delete bitmap cache size.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2219813859
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2220672061
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.3 seconds inserted 10000000 Rows, about 469K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "Yukang-Lian (via GitHub)" <gi...@apache.org>.
Yukang-Lian commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2220487110
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2220614070
<details>
<summary>TPC-H: <b>Total hot run time: 50116 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd, data reload: false
------ Round 1 ----------------------------------
q1 17822 4450 4459 4450
q2 2085 161 148 148
q3 10257 1927 1932 1927
q4 10118 1253 1328 1253
q5 8470 3962 3962 3962
q6 242 132 125 125
q7 2056 1607 1583 1583
q8 9382 2758 2738 2738
q9 11031 10493 10273 10273
q10 8671 3570 3549 3549
q11 424 237 248 237
q12 474 301 303 301
q13 18354 3950 4030 3950
q14 346 324 334 324
q15 528 456 462 456
q16 667 572 588 572
q17 1155 959 981 959
q18 7390 6906 6852 6852
q19 1839 1645 1603 1603
q20 588 312 297 297
q21 4482 4107 4127 4107
q22 541 456 450 450
Total cold run time: 116922 ms
Total hot run time: 50116 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4305 4290 4327 4290
q2 324 222 223 222
q3 4171 4136 4156 4136
q4 2763 2720 2758 2720
q5 7188 7118 7119 7118
q6 241 120 118 118
q7 3285 2871 2860 2860
q8 4418 4467 4487 4467
q9 16797 16798 16682 16682
q10 4227 4271 4258 4258
q11 716 718 691 691
q12 1041 838 842 838
q13 4412 3693 3741 3693
q14 452 438 432 432
q15 518 456 454 454
q16 729 682 673 673
q17 3814 3808 3835 3808
q18 8857 8697 8851 8697
q19 1753 1685 1674 1674
q20 2382 2182 2095 2095
q21 8479 8516 8554 8516
q22 1030 992 988 988
Total cold run time: 81902 ms
Total hot run time: 79430 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2226911144
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2219828027
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2220648593
<details>
<summary>TPC-DS: <b>Total hot run time: 203121 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd, data reload: false
query1 919 427 374 374
query2 6526 2654 2630 2630
query3 6922 203 199 199
query4 20499 18061 17956 17956
query5 19734 6508 6601 6508
query6 365 213 234 213
query7 4780 292 295 292
query8 457 430 415 415
query9 3135 2671 2608 2608
query10 429 300 300 300
query11 11449 10768 10797 10768
query12 121 80 74 74
query13 5600 730 706 706
query14 18945 13644 14054 13644
query15 369 241 258 241
query16 6451 288 263 263
query17 1618 1461 887 887
query18 2299 410 411 410
query19 221 150 154 150
query20 81 76 80 76
query21 192 96 94 94
query22 5228 5092 5023 5023
query23 32377 31965 32145 31965
query24 6987 6548 6517 6517
query25 524 436 442 436
query26 532 162 165 162
query27 1821 298 296 296
query28 6181 2360 2325 2325
query29 2883 2782 2773 2773
query30 243 163 168 163
query31 922 740 792 740
query32 70 65 61 61
query33 399 275 257 257
query34 844 469 472 469
query35 1155 934 876 876
query36 1332 1104 1330 1104
query37 90 63 60 60
query38 3054 2920 2909 2909
query39 1393 1314 1321 1314
query40 202 96 97 96
query41 49 45 44 44
query42 79 82 83 82
query43 679 685 686 685
query44 1118 724 720 720
query45 251 242 234 234
query46 1239 979 973 973
query47 1919 1751 1714 1714
query48 1020 717 724 717
query49 626 370 373 370
query50 863 605 642 605
query51 4763 4659 4696 4659
query52 97 81 79 79
query53 448 328 328 328
query54 2655 2479 2470 2470
query55 90 86 88 86
query56 247 217 221 217
query57 1272 1049 1227 1049
query58 224 223 198 198
query59 4215 3716 3735 3716
query60 231 203 197 197
query61 101 94 96 94
query62 883 454 473 454
query63 480 350 343 343
query64 2483 1514 1396 1396
query65 3618 3546 3566 3546
query66 794 381 381 381
query67 16172 16989 14873 14873
query68 8703 658 649 649
query69 582 361 350 350
query70 1627 1337 1344 1337
query71 414 311 318 311
query72 6529 3514 3487 3487
query73 733 324 321 321
query74 6480 5920 5881 5881
query75 5346 3611 3689 3611
query76 5220 1131 1121 1121
query77 863 272 264 264
query78 12788 11651 11577 11577
query79 8170 650 640 640
query80 1302 398 407 398
query81 488 236 237 236
query82 1699 95 102 95
query83 181 130 134 130
query84 256 72 70 70
query85 889 328 333 328
query86 339 291 306 291
query87 3220 3018 3000 3000
query88 4886 2303 2308 2303
query89 387 305 324 305
query90 1932 206 213 206
query91 197 139 140 139
query92 58 52 53 52
query93 5549 575 567 567
query94 713 211 204 204
query95 1129 1076 1077 1076
query96 652 323 321 321
query97 6473 6433 6259 6259
query98 205 181 172 172
query99 3023 836 864 836
Total cold run time: 314998 ms
Total hot run time: 203121 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2220663134
<details>
<summary>ClickBench: <b>Total hot run time: 30.62 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd, data reload: false
query1 0.02 0.02 0.03
query2 0.07 0.02 0.03
query3 0.25 0.05 0.05
query4 1.80 0.07 0.06
query5 0.53 0.52 0.52
query6 1.24 0.62 0.60
query7 0.02 0.01 0.01
query8 0.03 0.02 0.02
query9 0.52 0.49 0.48
query10 0.53 0.54 0.54
query11 0.12 0.09 0.09
query12 0.12 0.10 0.09
query13 0.62 0.60 0.61
query14 0.77 0.78 0.80
query15 0.76 0.75 0.76
query16 0.36 0.38 0.37
query17 0.97 1.02 1.03
query18 0.21 0.26 0.24
query19 1.95 1.87 1.89
query20 0.01 0.01 0.01
query21 15.49 0.53 0.55
query22 2.44 2.31 1.64
query23 17.31 0.90 0.95
query24 6.32 0.87 1.15
query25 0.36 0.10 0.04
query26 0.71 0.16 0.15
query27 0.05 0.04 0.05
query28 6.87 0.76 0.77
query29 12.72 2.35 2.24
query30 0.55 0.51 0.52
query31 2.84 0.38 0.37
query32 3.39 0.49 0.49
query33 3.12 3.04 3.07
query34 15.27 4.80 4.78
query35 4.87 4.85 4.84
query36 1.04 1.01 1.02
query37 0.06 0.05 0.04
query38 0.03 0.03 0.02
query39 0.02 0.01 0.02
query40 0.16 0.15 0.15
query41 0.07 0.02 0.02
query42 0.02 0.01 0.02
query43 0.02 0.02 0.02
Total cold run time: 104.63 s
Total hot run time: 30.62 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2226911154
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37602:
URL: https://github.com/apache/doris/pull/37602#issuecomment-2220650111
TeamCity be ut coverage result:
Function Coverage: 37.89% (8117/21420)
Line Coverage: 29.56% (66520/225016)
Region Coverage: 29.03% (34280/118096)
Branch Coverage: 24.89% (17607/70726)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd_f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #37602:
URL: https://github.com/apache/doris/pull/37602
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org