You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ByteYue (via GitHub)" <gi...@apache.org> on 2024/02/19 09:56:08 UTC
[PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
ByteYue opened a new pull request, #31117:
URL: https://github.com/apache/doris/pull/31117
## Proposed changes
Issue Number: close #xxx
In order to monitor whether the cooled rowset has been thoroughly reclaimed, a new boolean variable (bvar) has been introduced to track the count of all unused rowsets. This bvar may not be entirely accurate, as situations like push handlers can add unused rowsets too. However, it provides a rough indication of the reclamation status of cooled rowsets
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952493849
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952147209
<details>
<summary>TPC-H: <b>Total hot run time: 41339 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ccfed61d78bc27869894c12d7ed0dbb66b7dab81, data reload: false
------ Round 1 ----------------------------------
q1 17693 4942 4885 4885
q2 2033 140 131 131
q3 10586 1004 1005 1004
q4 4645 976 975 975
q5 7670 3236 3277 3236
q6 192 130 129 129
q7 1253 791 758 758
q8 9301 2069 2055 2055
q9 7553 6690 6672 6672
q10 8318 2639 2660 2639
q11 414 220 209 209
q12 769 331 334 331
q13 18000 3661 3679 3661
q14 285 262 263 262
q15 596 511 489 489
q16 481 402 410 402
q17 922 867 817 817
q18 7408 6580 6560 6560
q19 1545 1501 1504 1501
q20 634 344 344 344
q21 6382 3945 3975 3945
q22 863 339 334 334
Total cold run time: 107543 ms
Total hot run time: 41339 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4845 4844 4870 4844
q2 289 184 183 183
q3 3585 3562 3544 3544
q4 2540 2534 2529 2529
q5 5742 5752 5761 5752
q6 212 124 123 123
q7 2249 1667 1676 1667
q8 2996 3058 3104 3058
q9 8726 8702 8694 8694
q10 6771 4232 4230 4230
q11 520 364 373 364
q12 771 547 555 547
q13 4356 3406 3397 3397
q14 273 239 232 232
q15 597 496 527 496
q16 475 443 429 429
q17 1677 1607 1591 1591
q18 8315 7714 7539 7539
q19 1637 1648 1631 1631
q20 2084 1827 1823 1823
q21 6527 6171 6198 6171
q22 570 524 509 509
Total cold run time: 65757 ms
Total hot run time: 59353 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #31117:
URL: https://github.com/apache/doris/pull/31117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952106482
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952147039
TeamCity be ut coverage result:
Function Coverage: 35.77% (8557/23924)
Line Coverage: 27.72% (69392/250366)
Region Coverage: 26.84% (36008/134172)
Branch Coverage: 23.65% (18419/77884)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ccfed61d78bc27869894c12d7ed0dbb66b7dab81_ccfed61d78bc27869894c12d7ed0dbb66b7dab81/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952178246
<details>
<summary>ClickBench: <b>Total hot run time: 31.02 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ccfed61d78bc27869894c12d7ed0dbb66b7dab81, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.22 0.07 0.08
query4 1.67 0.08 0.08
query5 0.49 0.48 0.48
query6 1.36 0.61 0.63
query7 0.02 0.02 0.02
query8 0.04 0.02 0.02
query9 0.53 0.46 0.44
query10 0.49 0.48 0.50
query11 0.13 0.10 0.10
query12 0.12 0.10 0.10
query13 0.60 0.59 0.58
query14 0.76 0.79 0.79
query15 0.81 0.79 0.80
query16 0.34 0.34 0.33
query17 0.92 0.91 0.92
query18 0.16 0.17 0.19
query19 1.73 1.68 1.68
query20 0.02 0.01 0.01
query21 15.39 0.63 0.61
query22 3.08 3.88 2.17
query23 17.70 0.92 0.96
query24 2.06 0.38 0.69
query25 0.61 0.09 0.05
query26 0.17 0.14 0.15
query27 0.06 0.07 0.05
query28 11.77 0.84 0.84
query29 12.55 3.55 3.51
query30 0.57 0.50 0.48
query31 2.77 0.37 0.36
query32 3.35 0.47 0.48
query33 3.10 3.16 3.18
query34 15.36 4.48 4.50
query35 4.50 4.47 4.52
query36 1.07 0.95 0.96
query37 0.06 0.05 0.05
query38 0.03 0.04 0.03
query39 0.02 0.02 0.01
query40 0.18 0.14 0.14
query41 0.07 0.01 0.02
query42 0.03 0.02 0.02
query43 0.03 0.02 0.02
Total cold run time: 105.03 s
Total hot run time: 31.02 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952493791
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952168525
<details>
<summary>TPC-DS: <b>Total hot run time: 177678 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ccfed61d78bc27869894c12d7ed0dbb66b7dab81, data reload: false
query1 925 352 344 344
query2 6518 1791 1739 1739
query3 6689 209 203 203
query4 22996 21099 21240 21099
query5 4267 466 375 375
query6 250 169 172 169
query7 4880 317 298 298
query8 251 200 199 199
query9 8728 2852 2838 2838
query10 417 236 230 230
query11 15085 14709 14533 14533
query12 144 83 89 83
query13 1709 433 433 433
query14 9632 7767 7699 7699
query15 218 190 191 190
query16 7716 278 254 254
query17 2075 776 588 588
query18 3535 311 298 298
query19 453 200 179 179
query20 177 95 111 95
query21 441 152 158 152
query22 5874 5336 4621 4621
query23 32867 31506 31536 31506
query24 12452 3396 3397 3396
query25 642 374 361 361
query26 1842 159 160 159
query27 3040 310 316 310
query28 6514 1829 1822 1822
query29 1085 603 618 603
query30 269 137 146 137
query31 914 742 773 742
query32 97 55 57 55
query33 719 246 241 241
query34 1079 494 505 494
query35 946 835 830 830
query36 963 894 866 866
query37 137 59 67 59
query38 3273 3184 3177 3177
query39 1378 1331 1317 1317
query40 282 104 106 104
query41 37 35 36 35
query42 109 100 99 99
query43 470 467 453 453
query44 1055 684 699 684
query45 197 182 180 180
query46 1063 786 758 758
query47 1693 1536 1547 1536
query48 409 356 350 350
query49 1164 303 295 295
query50 774 364 363 363
query51 5307 5221 5139 5139
query52 113 95 91 91
query53 384 297 295 295
query54 296 215 230 215
query55 81 84 79 79
query56 221 193 204 193
query57 1072 901 955 901
query58 204 196 197 196
query59 2282 2150 2141 2141
query60 238 221 220 220
query61 85 84 87 84
query62 599 368 391 368
query63 341 285 283 283
query64 6274 3044 3148 3044
query65 3299 3253 3255 3253
query66 1340 328 318 318
query67 14568 14362 14161 14161
query68 5137 566 566 566
query69 524 355 361 355
query70 1271 1228 1214 1214
query71 444 254 251 251
query72 6346 2765 2638 2638
query73 703 310 329 310
query74 6831 6453 6391 6391
query75 3240 2578 2538 2538
query76 3351 1143 1200 1143
query77 361 235 236 235
query78 9423 8830 8752 8752
query79 961 509 517 509
query80 512 349 335 335
query81 441 205 205 205
query82 166 88 82 82
query83 142 122 121 121
query84 228 78 77 77
query85 1022 348 341 341
query86 298 299 305 299
query87 3467 3350 3291 3291
query88 2708 2288 2279 2279
query89 439 359 362 359
query90 1998 168 165 165
query91 159 132 128 128
query92 55 50 46 46
query93 1010 520 482 482
query94 1279 182 180 180
query95 8608 8568 371 371
query96 585 266 272 266
query97 4490 4282 4301 4282
query98 216 200 204 200
query99 1026 735 768 735
Total cold run time: 282595 ms
Total hot run time: 177678 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952087486
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "ByteYue (via GitHub)" <gi...@apache.org>.
ByteYue commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952089280
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (enhance)(Cooldown)Add bvar to monitor number of the unused rowsets [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31117:
URL: https://github.com/apache/doris/pull/31117#issuecomment-1952183743
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit ccfed61d78bc27869894c12d7ed0dbb66b7dab81 with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 13.5 seconds inserted 10000000 Rows, about 740K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org