You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Lchangliang (via GitHub)" <gi...@apache.org> on 2024/07/10 07:40:23 UTC
[PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang opened a new pull request, #37597:
URL: https://github.com/apache/doris/pull/37597
https://github.com/apache/doris/pull/34526
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2219897355
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2219916272
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2220033023
<details>
<summary>TPC-H: <b>Total hot run time: 50232 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1aeff68efc9a98a8c9feb11736bfa44ede739476, data reload: false
------ Round 1 ----------------------------------
q1 17868 4424 4345 4345
q2 2092 163 145 145
q3 10420 1944 1960 1944
q4 10301 1279 1344 1279
q5 8503 3951 3994 3951
q6 229 128 127 127
q7 2057 1623 1623 1623
q8 9307 2777 2726 2726
q9 10684 10494 10305 10305
q10 8656 3572 3523 3523
q11 428 247 246 246
q12 480 302 310 302
q13 18379 4015 4077 4015
q14 360 328 336 328
q15 496 460 457 457
q16 695 565 576 565
q17 1160 971 984 971
q18 7400 6941 6886 6886
q19 1779 1674 1641 1641
q20 553 308 312 308
q21 4484 4100 4125 4100
q22 532 445 445 445
Total cold run time: 116863 ms
Total hot run time: 50232 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4349 4286 4317 4286
q2 322 223 229 223
q3 4212 4175 4156 4156
q4 2765 2742 2766 2742
q5 7217 7122 7100 7100
q6 239 120 120 120
q7 3289 2821 2904 2821
q8 4400 4482 4470 4470
q9 16990 16875 16747 16747
q10 4199 4269 4266 4266
q11 739 702 696 696
q12 1040 849 864 849
q13 6813 3732 3706 3706
q14 457 439 423 423
q15 523 464 462 462
q16 722 686 705 686
q17 3876 3839 3864 3839
q18 8855 8770 8797 8770
q19 1733 1742 1678 1678
q20 2416 2171 2119 2119
q21 8617 8506 8507 8506
q22 1047 1001 995 995
Total cold run time: 84820 ms
Total hot run time: 79660 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2219780603
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2220075157
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 1aeff68efc9a98a8c9feb11736bfa44ede739476 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2220031153
TeamCity be ut coverage result:
Function Coverage: 37.88% (8114/21420)
Line Coverage: 29.55% (66488/225006)
Region Coverage: 29.02% (34264/118085)
Branch Coverage: 24.89% (17599/70716)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1aeff68efc9a98a8c9feb11736bfa44ede739476_1aeff68efc9a98a8c9feb11736bfa44ede739476/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2220068379
<details>
<summary>ClickBench: <b>Total hot run time: 31.05 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1aeff68efc9a98a8c9feb11736bfa44ede739476, data reload: false
query1 0.03 0.03 0.02
query2 0.07 0.03 0.02
query3 0.25 0.05 0.04
query4 1.79 0.07 0.06
query5 0.54 0.52 0.52
query6 1.23 0.61 0.61
query7 0.01 0.01 0.02
query8 0.03 0.03 0.02
query9 0.52 0.48 0.49
query10 0.54 0.53 0.53
query11 0.12 0.08 0.09
query12 0.11 0.09 0.09
query13 0.63 0.60 0.61
query14 0.81 0.78 0.79
query15 0.78 0.77 0.76
query16 0.35 0.37 0.36
query17 1.02 1.01 0.97
query18 0.24 0.25 0.24
query19 1.91 1.89 1.81
query20 0.01 0.01 0.00
query21 15.49 0.56 0.54
query22 1.95 2.01 1.70
query23 17.27 0.90 0.89
query24 7.29 1.29 1.29
query25 0.40 0.11 0.06
query26 0.77 0.18 0.15
query27 0.04 0.05 0.03
query28 5.28 0.75 0.71
query29 12.65 2.38 2.37
query30 0.58 0.49 0.55
query31 2.80 0.39 0.38
query32 3.35 0.50 0.49
query33 3.14 3.04 3.12
query34 15.24 4.80 4.80
query35 4.86 4.87 4.83
query36 1.09 1.02 1.02
query37 0.05 0.04 0.05
query38 0.04 0.02 0.02
query39 0.02 0.02 0.02
query40 0.16 0.14 0.14
query41 0.07 0.01 0.02
query42 0.02 0.02 0.01
query43 0.02 0.02 0.02
Total cold run time: 103.57 s
Total hot run time: 31.05 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2219793011
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2219780561
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2219901833
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37597:
URL: https://github.com/apache/doris/pull/37597#issuecomment-2220057619
<details>
<summary>TPC-DS: <b>Total hot run time: 205563 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1aeff68efc9a98a8c9feb11736bfa44ede739476, data reload: false
query1 944 417 386 386
query2 6542 2863 2747 2747
query3 6922 212 210 210
query4 21310 17987 17945 17945
query5 19729 6501 6569 6501
query6 273 228 246 228
query7 4161 306 304 304
query8 452 515 401 401
query9 3147 2739 2672 2672
query10 416 320 297 297
query11 11299 10755 10763 10755
query12 124 82 76 76
query13 5603 680 695 680
query14 18124 13286 13402 13286
query15 360 241 259 241
query16 6472 286 262 262
query17 1740 1442 875 875
query18 2330 420 417 417
query19 212 145 151 145
query20 81 82 80 80
query21 191 96 103 96
query22 5303 5051 5088 5051
query23 32537 31975 31897 31897
query24 6914 6498 6544 6498
query25 516 431 416 416
query26 526 164 159 159
query27 1839 292 300 292
query28 6198 2389 2330 2330
query29 3020 2773 2767 2767
query30 241 162 171 162
query31 906 722 740 722
query32 73 63 61 61
query33 406 263 261 261
query34 850 501 474 474
query35 1141 917 939 917
query36 1280 1165 1084 1084
query37 90 59 60 59
query38 3065 2938 2897 2897
query39 1381 1323 1314 1314
query40 203 97 96 96
query41 48 48 45 45
query42 86 83 90 83
query43 822 738 738 738
query44 1129 738 727 727
query45 253 241 237 237
query46 1231 959 976 959
query47 1782 1680 1748 1680
query48 1024 737 700 700
query49 633 380 380 380
query50 858 586 635 586
query51 4774 4717 4717 4717
query52 104 80 92 80
query53 458 332 322 322
query54 2647 2460 2468 2460
query55 99 75 86 75
query56 249 233 208 208
query57 1174 1102 1100 1100
query58 223 211 199 199
query59 4294 4187 3999 3999
query60 217 224 208 208
query61 100 95 100 95
query62 702 445 472 445
query63 496 345 347 345
query64 2399 1553 1537 1537
query65 3625 3556 3574 3556
query66 763 382 398 382
query67 17468 18531 16991 16991
query68 5563 659 664 659
query69 558 365 370 365
query70 1406 1291 1512 1291
query71 367 314 320 314
query72 6358 3493 3487 3487
query73 715 334 331 331
query74 6273 5900 5829 5829
query75 4473 3693 3609 3609
query76 3233 1160 1183 1160
query77 568 264 275 264
query78 12394 11635 11805 11635
query79 7064 617 657 617
query80 2975 402 417 402
query81 540 240 231 231
query82 1620 95 97 95
query83 315 138 140 138
query84 261 72 71 71
query85 1495 334 337 334
query86 390 303 311 303
query87 3290 3018 3062 3018
query88 4826 2422 2400 2400
query89 366 307 292 292
query90 1636 220 206 206
query91 190 157 143 143
query92 64 63 53 53
query93 2287 550 563 550
query94 900 212 205 205
query95 1128 1078 1081 1078
query96 650 331 344 331
query97 6527 6450 6267 6267
query98 195 180 181 180
query99 3127 863 790 790
Total cold run time: 306333 ms
Total hot run time: 205563 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring closed pull request #37597: [Improve] Estimated column reader memory to control segment cache
URL: https://github.com/apache/doris/pull/37597
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org