You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "morningman (via GitHub)" <gi...@apache.org> on 2024/03/18 03:39:58 UTC
[PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
morningman opened a new pull request, #32367:
URL: https://github.com/apache/doris/pull/32367
bp #32340
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #32367:
URL: https://github.com/apache/doris/pull/32367
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002923887
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 9e31295ddf214ffe330a7353dca15e7351a94188 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.0 seconds inserted 10000000 Rows, about 476K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002917480
<details>
<summary>TPC-DS: <b>Total hot run time: 201975 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e31295ddf214ffe330a7353dca15e7351a94188, data reload: false
query1 922 394 383 383
query2 6526 2153 2056 2056
query3 6914 197 198 197
query4 21255 18161 18129 18129
query5 20433 6582 6519 6519
query6 287 205 241 205
query7 4154 296 297 296
query8 264 235 246 235
query9 3245 2794 2694 2694
query10 430 289 302 289
query11 11419 10823 11169 10823
query12 125 81 81 81
query13 5617 672 663 663
query14 18950 13821 13399 13399
query15 363 232 238 232
query16 7512 294 277 277
query17 3167 2375 889 889
query18 3342 404 400 400
query19 202 143 146 143
query20 80 76 77 76
query21 184 89 91 89
query22 5319 5138 4898 4898
query23 32608 31868 31891 31868
query24 6891 6542 6492 6492
query25 504 421 420 420
query26 472 161 154 154
query27 1683 293 297 293
query28 6089 2242 2195 2195
query29 2800 2718 2775 2718
query30 242 161 159 159
query31 901 713 717 713
query32 66 56 60 56
query33 387 245 242 242
query34 839 457 478 457
query35 1114 947 897 897
query36 1686 1419 1565 1419
query37 90 60 57 57
query38 3093 2875 2905 2875
query39 1364 1312 1329 1312
query40 198 90 94 90
query41 35 33 32 32
query42 91 82 81 81
query43 666 642 555 555
query44 1149 718 724 718
query45 241 223 228 223
query46 1243 971 963 963
query47 1823 1674 1687 1674
query48 974 672 685 672
query49 605 366 370 366
query50 861 590 618 590
query51 4772 4730 4675 4675
query52 87 77 72 72
query53 457 319 319 319
query54 2675 2453 2438 2438
query55 92 73 79 73
query56 215 197 206 197
query57 1178 1095 1133 1095
query58 214 202 190 190
query59 3386 3226 3388 3226
query60 194 184 205 184
query61 83 79 79 79
query62 734 540 453 453
query63 491 338 343 338
query64 3554 1538 1374 1374
query65 3621 3519 3566 3519
query66 617 366 357 357
query67 15888 15369 15893 15369
query68 8402 673 675 673
query69 560 363 347 347
query70 2119 1742 1850 1742
query71 407 299 295 295
query72 6426 3427 3421 3421
query73 741 330 328 328
query74 6259 5848 5803 5803
query75 4578 3647 3520 3520
query76 4669 1184 1209 1184
query77 566 242 243 242
query78 12508 11705 11608 11608
query79 7563 630 634 630
query80 1340 379 396 379
query81 500 232 225 225
query82 1516 92 96 92
query83 158 134 132 132
query84 255 68 66 66
query85 958 276 276 276
query86 417 385 400 385
query87 3229 3032 3076 3032
query88 4979 2326 2327 2326
query89 367 272 277 272
query90 1839 211 210 210
query91 148 115 123 115
query92 55 51 45 45
query93 3950 586 561 561
query94 802 201 205 201
query95 1112 1063 1060 1060
query96 630 329 330 329
query97 6478 6223 6303 6223
query98 192 163 173 163
query99 2999 868 911 868
Total cold run time: 314405 ms
Total hot run time: 201975 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002877819
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002879880
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002907604
<details>
<summary>TPC-H: <b>Total hot run time: 49512 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e31295ddf214ffe330a7353dca15e7351a94188, data reload: false
------ Round 1 ----------------------------------
q1 17685 4374 4322 4322
q2 2026 148 137 137
q3 10460 1891 1951 1891
q4 10380 1250 1311 1250
q5 8410 4068 3902 3902
q6 230 122 124 122
q7 2034 1604 1621 1604
q8 9456 2694 2696 2694
q9 13415 10459 10376 10376
q10 8640 3498 3484 3484
q11 414 243 246 243
q12 464 299 300 299
q13 18338 3907 4012 3907
q14 352 326 326 326
q15 498 455 454 454
q16 681 593 588 588
q17 1114 924 947 924
q18 7238 6683 6900 6683
q19 1676 1607 1555 1555
q20 547 303 304 303
q21 4453 4068 4076 4068
q22 500 380 393 380
Total cold run time: 119011 ms
Total hot run time: 49512 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4334 4284 4303 4284
q2 315 217 219 217
q3 4221 4139 4130 4130
q4 2762 2745 2722 2722
q5 7205 7171 7150 7150
q6 235 124 117 117
q7 3282 2835 2786 2786
q8 4349 4438 4478 4438
q9 17068 16983 16915 16915
q10 4243 4263 4272 4263
q11 752 695 654 654
q12 1015 842 852 842
q13 7030 3717 3758 3717
q14 457 432 415 415
q15 500 455 450 450
q16 756 691 687 687
q17 3873 3909 3840 3840
q18 8697 8725 8744 8725
q19 1743 1691 1709 1691
q20 2391 2232 2151 2151
q21 8673 8801 8827 8801
q22 1033 963 962 962
Total cold run time: 84934 ms
Total hot run time: 79957 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002908017
TeamCity be ut coverage result:
Function Coverage: 37.81% (8040/21265)
Line Coverage: 29.48% (65609/222558)
Region Coverage: 28.95% (33760/116621)
Branch Coverage: 24.79% (17327/69890)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9e31295ddf214ffe330a7353dca15e7351a94188_9e31295ddf214ffe330a7353dca15e7351a94188/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002921436
<details>
<summary>ClickBench: <b>Total hot run time: 30.6 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9e31295ddf214ffe330a7353dca15e7351a94188, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.25 0.05 0.06
query4 1.79 0.08 0.07
query5 0.54 0.51 0.52
query6 1.23 0.62 0.62
query7 0.01 0.01 0.01
query8 0.03 0.02 0.03
query9 0.54 0.48 0.49
query10 0.55 0.54 0.53
query11 0.11 0.09 0.08
query12 0.11 0.10 0.09
query13 0.62 0.62 0.61
query14 0.80 0.78 0.81
query15 0.78 0.76 0.78
query16 0.38 0.38 0.36
query17 1.00 1.03 1.00
query18 0.25 0.25 0.25
query19 1.93 1.88 1.85
query20 0.01 0.01 0.01
query21 15.46 0.58 0.56
query22 1.84 2.21 1.10
query23 17.27 1.04 0.93
query24 6.00 1.31 1.35
query25 0.35 0.10 0.05
query26 0.77 0.16 0.17
query27 0.05 0.04 0.04
query28 6.98 0.73 0.72
query29 12.71 2.32 2.34
query30 0.61 0.49 0.49
query31 2.82 0.40 0.37
query32 3.39 0.50 0.49
query33 3.06 3.06 3.06
query34 15.26 4.82 4.78
query35 4.84 4.82 4.83
query36 1.06 1.01 1.03
query37 0.06 0.05 0.05
query38 0.04 0.02 0.02
query39 0.02 0.02 0.01
query40 0.16 0.14 0.14
query41 0.07 0.01 0.01
query42 0.02 0.02 0.02
query43 0.02 0.02 0.02
Total cold run time: 103.88 s
Total hot run time: 30.6 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002884930
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002871793
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002873796
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file-meta-cache) reduce file meta cache size and disable cache for some cases (#32340) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32367:
URL: https://github.com/apache/doris/pull/32367#issuecomment-2002889909
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org