You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zzzxl1993 (via GitHub)" <gi...@apache.org> on 2024/01/04 09:40:39 UTC
[PR] (opt)(file cache) make the cache queue percentage configurable [doris]
zzzxl1993 opened a new pull request, #29537:
URL: https://github.com/apache/doris/pull/29537
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1880480715
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #29537:
URL: https://github.com/apache/doris/pull/29537#discussion_r1441659633
##########
be/src/common/config.cpp:
##########
@@ -999,6 +999,7 @@ DEFINE_Validator(file_cache_min_file_segment_size, [](const int64_t config) -> b
DEFINE_Bool(clear_file_cache, "false");
DEFINE_Bool(enable_file_cache_query_limit, "false");
DEFINE_mInt32(file_cache_wait_sec_after_fail, "0"); // // zero for no waiting and retrying
+DEFINE_String(file_cache_queue_percentage, "[17,2,1,20]");
Review Comment:
suggest name file_cache_queue_percent
##########
be/src/common/config.cpp:
##########
@@ -999,6 +999,7 @@ DEFINE_Validator(file_cache_min_file_segment_size, [](const int64_t config) -> b
DEFINE_Bool(clear_file_cache, "false");
DEFINE_Bool(enable_file_cache_query_limit, "false");
DEFINE_mInt32(file_cache_wait_sec_after_fail, "0"); // // zero for no waiting and retrying
+DEFINE_String(file_cache_queue_percentage, "[17,2,1,20]");
Review Comment:
17 + 2 + 1 + 20 < 100
##########
be/src/common/config.cpp:
##########
@@ -999,6 +999,7 @@ DEFINE_Validator(file_cache_min_file_segment_size, [](const int64_t config) -> b
DEFINE_Bool(clear_file_cache, "false");
DEFINE_Bool(enable_file_cache_query_limit, "false");
DEFINE_mInt32(file_cache_wait_sec_after_fail, "0"); // // zero for no waiting and retrying
+DEFINE_String(file_cache_queue_percentage, "[17,2,1,20]");
Review Comment:
add doc for be config
##########
be/src/olap/options.cpp:
##########
@@ -244,18 +244,44 @@ Status parse_conf_cache_paths(const std::string& config_path, std::vector<CacheP
return Status::OK();
}
-io::FileCacheSettings CachePath::init_settings() const {
+Status parse_conf_cache_queue_percentage(std::string percentage_config,
+ std::vector<size_t>& percentages) {
+ if (percentage_config.front() != '[' || percentage_config.back() != ']') {
+ return Status::InvalidArgument("wrong percentage format");
Review Comment:
add more info for valid format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1880379861
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878638234
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878452056
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit e6da27ac2fe876f645d95825181674f36ffd7491, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5432 5194 5123 5123
q2 386 187 151 151
q3 1454 1126 1096 1096
q4 1077 819 726 726
q5 3126 3104 3129 3104
q6 221 133 135 133
q7 966 520 547 520
q8 2155 2181 2201 2181
q9 6695 6670 6660 6660
q10 3204 3142 3142 3142
q11 336 213 225 213
q12 378 239 231 231
q13 4419 3631 3594 3594
q14 245 220 215 215
q15 580 538 531 531
q16 463 411 406 406
q17 1049 549 510 510
q18 7060 6716 6874 6716
q19 1641 1523 1388 1388
q20 562 327 358 327
q21 2891 2465 2488 2465
q22 386 317 326 317
Total cold run time: 44726 ms
Total hot run time: 39749 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5086 5075 5051 5051
q2 340 229 248 229
q3 3378 3354 3317 3317
q4 2159 2050 2069 2050
q5 5952 5904 5901 5901
q6 226 126 126 126
q7 2375 1941 1942 1941
q8 3552 3642 3679 3642
q9 9066 9002 9004 9002
q10 3860 3903 3908 3903
q11 568 474 461 461
q12 806 630 617 617
q13 3849 3191 3172 3172
q14 289 263 268 263
q15 594 520 522 520
q16 555 486 510 486
q17 2022 1811 1773 1773
q18 8764 8312 8327 8312
q19 1763 1679 1725 1679
q20 2255 1990 1964 1964
q21 5774 5298 5250 5250
q22 537 466 480 466
Total cold run time: 63770 ms
Total hot run time: 60125 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877314600
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit acbb64ff690dc4f827a4c8dc8765253a86d3070c, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 934 351 330 330
query2 6441 1930 1838 1838
query3 6655 214 201 201
query4 26284 22491 22410 22410
query5 4358 543 514 514
query6 274 182 192 182
query7 4598 280 261 261
query8 230 204 198 198
query9 8265 2569 2496 2496
query10 483 283 247 247
query11 16291 15531 15534 15531
query12 127 78 76 76
query13 1632 317 323 317
query14 11526 7060 7133 7060
query15 218 199 188 188
query16 6429 270 262 262
query17 1844 499 486 486
query18 1952 271 274 271
query19 206 136 133 133
query20 80 75 77 75
query21 183 96 96 96
query22 4964 4753 4848 4753
query23 32254 31282 31183 31183
query24 12096 2824 2836 2824
query25 597 354 347 347
query26 1748 141 134 134
query27 2924 266 271 266
query28 7028 1866 1863 1863
query29 2065 387 386 386
query30 285 142 149 142
query31 961 769 767 767
query32 90 56 52 52
query33 751 289 257 257
query34 884 442 436 436
query35 891 774 750 750
query36 1305 1109 1255 1109
query37 186 71 65 65
query38 3371 3300 3237 3237
query39 1329 1286 1269 1269
query40 314 90 88 88
query41 38 43 35 35
query42 88 83 84 83
query43 508 505 480 480
query44 1045 686 699 686
query45 196 188 184 184
query46 1064 642 638 638
query47 1645 1553 1529 1529
query48 321 257 254 254
query49 1215 323 326 323
query50 748 318 320 318
query51 5326 5279 5391 5279
query52 96 88 85 85
query53 206 144 143 143
query54 1414 553 584 553
query55 98 83 78 78
query56 186 187 192 187
query57 1020 913 925 913
query58 222 200 203 200
query59 2764 2613 2586 2586
query60 232 199 218 199
query61 88 85 86 85
query62 646 461 469 461
query63 156 145 141 141
query64 5916 1749 1743 1743
query65 3300 3283 3249 3249
query66 1392 353 331 331
query67 15597 15318 14939 14939
query68 11354 534 499 499
query69 514 255 244 244
query70 1712 1489 1457 1457
query71 490 222 226 222
query72 5647 3594 3543 3543
query73 2374 316 303 303
query74 7062 6532 6479 6479
query75 4807 2252 2285 2252
query76 6238 1111 1102 1102
query77 658 240 256 240
query78 9070 8642 8726 8642
query79 1040 498 485 485
query80 542 347 347 347
query81 460 206 210 206
query82 205 93 90 90
query83 174 136 136 136
query84 246 54 53 53
query85 958 278 270 270
query86 383 386 413 386
query87 3533 3463 3447 3447
query88 3008 2228 2233 2228
query89 325 262 251 251
query90 1890 200 190 190
query91 118 87 99 87
query92 56 45 52 45
query93 1198 493 421 421
query94 825 179 170 170
query95 434 415 393 393
query96 611 316 317 316
query97 4248 4155 4212 4155
query98 201 196 190 190
query99 1115 775 797 775
Total cold run time: 290076 ms
Total hot run time: 178426 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878734646
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit b57209b85901d160626a14e95ac8310692dbb07c, data reload: false
------ Round 1 ----------------------------------
q1 17815 5429 5145 5145
q2 2025 151 140 140
q3 10616 1067 1042 1042
q4 10199 737 759 737
q5 7823 2923 2963 2923
q6 216 131 131 131
q7 916 511 540 511
q8 9311 2025 2038 2025
q9 6868 6407 6358 6358
q10 8258 3066 3003 3003
q11 430 221 208 208
q12 393 233 235 233
q13 17996 3643 3630 3630
q14 245 221 210 210
q15 554 519 514 514
q16 445 397 406 397
q17 966 452 528 452
q18 7410 6742 6737 6737
q19 1594 1349 1385 1349
q20 718 330 344 330
q21 2834 2389 2393 2389
q22 381 326 339 326
Total cold run time: 108013 ms
Total hot run time: 38790 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5162 5166 5065 5065
q2 342 233 256 233
q3 3313 3286 3261 3261
q4 2143 2048 2035 2035
q5 5812 5774 5816 5774
q6 216 126 124 124
q7 2324 1926 1915 1915
q8 3374 3506 3493 3493
q9 8814 8753 8727 8727
q10 3778 3836 3845 3836
q11 585 491 482 482
q12 788 632 641 632
q13 6707 3196 3190 3190
q14 293 283 295 283
q15 569 511 516 511
q16 569 485 503 485
q17 1931 1768 1779 1768
q18 8714 8420 8253 8253
q19 1636 1619 1553 1553
q20 2177 1975 1954 1954
q21 5534 5267 5329 5267
q22 550 512 502 502
Total cold run time: 65331 ms
Total hot run time: 59343 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877091816
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 738cea3bf39cd23b4ef8505cb42c393fbefffb82, data reload: false
------ Round 1 ----------------------------------
q1 17672 5288 5109 5109
q2 2017 154 145 145
q3 10532 1045 1094 1045
q4 10181 732 833 732
q5 7804 2993 2939 2939
q6 211 132 130 130
q7 936 510 489 489
q8 9273 2009 2003 2003
q9 6876 6417 6381 6381
q10 8254 3031 3099 3031
q11 437 231 212 212
q12 388 224 231 224
q13 18013 3645 3619 3619
q14 248 225 217 217
q15 560 531 513 513
q16 455 405 427 405
q17 960 515 487 487
q18 7311 6719 6711 6711
q19 1595 1326 1424 1326
q20 694 317 322 317
q21 2790 2314 2393 2314
q22 366 336 310 310
Total cold run time: 107573 ms
Total hot run time: 38659 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5128 5192 5124 5124
q2 336 230 238 230
q3 3285 3290 3254 3254
q4 2132 1994 2030 1994
q5 5802 5788 5776 5776
q6 217 125 124 124
q7 2319 1871 1879 1871
q8 3358 3469 3493 3469
q9 8824 8754 8759 8754
q10 3788 3818 3835 3818
q11 569 483 481 481
q12 779 630 637 630
q13 7877 3203 3172 3172
q14 296 277 303 277
q15 584 513 520 513
q16 556 487 513 487
q17 1971 1782 1738 1738
q18 8698 8321 8220 8220
q19 1623 1609 1555 1555
q20 2183 1980 1931 1931
q21 5637 5274 5253 5253
q22 552 504 467 467
Total cold run time: 66514 ms
Total hot run time: 59138 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878260705
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1876894369
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 1e38fef227a5186bb4dbb63fe11b97b5e27ab93b, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 937 351 333 333
query2 6427 2108 1904 1904
query3 6654 202 202 202
query4 26490 22458 22401 22401
query5 3901 500 541 500
query6 257 180 182 180
query7 4574 268 269 268
query8 240 204 197 197
query9 8290 2599 2563 2563
query10 448 248 251 248
query11 16304 15588 15546 15546
query12 139 78 72 72
query13 1628 325 321 321
query14 11562 7169 7112 7112
query15 223 185 182 182
query16 6247 274 259 259
query17 1850 483 497 483
query18 1877 269 256 256
query19 182 138 129 129
query20 77 69 72 69
query21 180 97 95 95
query22 4985 4820 4833 4820
query23 32139 31358 31090 31090
query24 12196 2847 2804 2804
query25 598 338 327 327
query26 1807 142 141 141
query27 2918 274 276 274
query28 6987 1922 1944 1922
query29 2216 397 375 375
query30 282 140 147 140
query31 977 762 762 762
query32 93 53 60 53
query33 745 274 270 270
query34 875 450 439 439
query35 864 771 771 771
query36 1316 1241 1183 1183
query37 183 71 67 67
query38 3361 3241 3310 3241
query39 1314 1284 1289 1284
query40 299 86 83 83
query41 39 35 34 34
query42 84 88 85 85
query43 514 509 490 490
query44 1110 706 703 703
query45 192 186 181 181
query46 1065 643 642 642
query47 1673 1542 1631 1542
query48 335 254 259 254
query49 1212 304 322 304
query50 726 365 327 327
query51 5420 5365 5248 5248
query52 94 90 89 89
query53 209 150 145 145
query54 1414 554 574 554
query55 97 80 78 78
query56 197 179 197 179
query57 1057 942 937 937
query58 218 202 195 195
query59 2874 2722 2632 2632
query60 228 199 216 199
query61 88 81 99 81
query62 650 464 471 464
query63 166 148 149 148
query64 5810 1672 1651 1651
query65 3333 3246 3262 3246
query66 1398 327 335 327
query67 15586 15593 15356 15356
query68 9749 523 540 523
query69 502 243 254 243
query70 1770 1506 1545 1506
query71 462 217 214 214
query72 5033 3497 3528 3497
query73 1931 309 313 309
query74 6938 6427 6471 6427
query75 4546 2261 2275 2261
query76 6081 1112 1096 1096
query77 637 243 266 243
query78 9128 8862 8580 8580
query79 1042 509 496 496
query80 732 353 330 330
query81 455 204 205 204
query82 205 96 104 96
query83 184 134 134 134
query84 251 53 53 53
query85 921 277 269 269
query86 386 404 395 395
query87 3508 3363 3360 3360
query88 2817 2242 2240 2240
query89 332 255 255 255
query90 1891 187 192 187
query91 116 93 91 91
query92 60 49 48 48
query93 1030 502 427 427
query94 789 181 179 179
query95 450 406 417 406
query96 610 320 310 310
query97 4288 4154 4183 4154
query98 214 196 186 186
query99 1127 868 847 847
Total cold run time: 286616 ms
Total hot run time: 179023 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877012436
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878323415
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit ac22fd81f29416d6a683e4a835628e5e4c4084ee, data reload: false
------ Round 1 ----------------------------------
q1 18555 5200 5158 5158
q2 2026 167 146 146
q3 10599 1110 1198 1110
q4 10203 793 777 777
q5 7789 2988 2942 2942
q6 211 131 132 131
q7 917 514 531 514
q8 9295 2045 1982 1982
q9 6877 6457 6430 6430
q10 8213 3105 2985 2985
q11 430 214 209 209
q12 391 232 233 232
q13 18025 3653 3661 3653
q14 250 217 216 216
q15 571 526 512 512
q16 450 393 393 393
q17 985 543 551 543
q18 7413 6739 6633 6633
q19 1590 1343 1343 1343
q20 736 339 338 338
q21 2818 2337 2446 2337
q22 382 322 333 322
Total cold run time: 108726 ms
Total hot run time: 38906 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5127 5110 5119 5110
q2 339 227 244 227
q3 3302 3264 3278 3264
q4 2129 1998 2012 1998
q5 5852 5812 5820 5812
q6 219 122 123 122
q7 2341 1929 1901 1901
q8 3429 3474 3460 3460
q9 8864 8779 8790 8779
q10 3811 3866 3849 3849
q11 568 485 491 485
q12 798 643 645 643
q13 6853 3212 3177 3177
q14 290 274 268 268
q15 589 521 527 521
q16 547 497 508 497
q17 1936 1796 1834 1796
q18 8834 8318 8241 8241
q19 1639 1575 1595 1575
q20 2222 1981 1943 1943
q21 5650 5230 5306 5230
q22 609 462 485 462
Total cold run time: 65948 ms
Total hot run time: 59360 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878925538
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.28 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.8 seconds inserted 10000000 Rows, about 359K ops/s
storage size: 17184131830 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878845174
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1876800688
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877205652
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878103342
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 8d19aa883733a3766de71ea2b13f716b1018a658, data reload: false
------ Round 1 ----------------------------------
q1 17703 5082 5009 5009
q2 2038 155 144 144
q3 10538 1111 1137 1111
q4 10182 795 823 795
q5 7789 2961 2927 2927
q6 215 133 135 133
q7 926 526 534 526
q8 9242 2033 2030 2030
q9 6880 6409 6380 6380
q10 8257 3013 3009 3009
q11 421 217 223 217
q12 411 237 231 231
q13 18011 3669 3667 3667
q14 244 215 223 215
q15 568 528 519 519
q16 461 401 392 392
q17 970 530 536 530
q18 7348 6763 6741 6741
q19 1598 1384 1394 1384
q20 745 360 328 328
q21 2861 2438 2443 2438
q22 372 318 343 318
Total cold run time: 107780 ms
Total hot run time: 39044 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5173 5072 5040 5040
q2 336 227 237 227
q3 3291 3298 3269 3269
q4 2154 1995 1987 1987
q5 5766 5747 5789 5747
q6 213 128 128 128
q7 2323 1930 1881 1881
q8 3364 3430 3487 3430
q9 8815 8755 8739 8739
q10 3785 3800 3818 3800
q11 580 482 503 482
q12 801 641 628 628
q13 7285 3246 3245 3245
q14 289 252 274 252
q15 577 520 521 520
q16 551 492 486 486
q17 1928 1749 1758 1749
q18 8721 8431 8341 8341
q19 1631 1577 1585 1577
q20 2220 1956 1954 1954
q21 5735 5391 5331 5331
q22 557 515 479 479
Total cold run time: 66095 ms
Total hot run time: 59292 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang commented on code in PR #29537:
URL: https://github.com/apache/doris/pull/29537#discussion_r1443600889
##########
be/src/olap/options.cpp:
##########
@@ -235,7 +238,38 @@ Status parse_conf_cache_paths(const std::string& config_path, std::vector<CacheP
return Status::InvalidArgument(
"total_size or query_limit should not less than or equal to zero");
}
- paths.emplace_back(std::move(path), total_size, query_limit_bytes);
+
+ // percent
+ auto get_percent_value = [&](const std::string& key, size_t& percent) {
+ auto& value = map.FindMember(key.c_str())->value;
+ if (value.IsUint()) {
+ percent = value.GetUint();
+ } else {
+ return Status::InvalidArgument("percent should be uint");
+ }
+ return Status::OK();
+ };
+
+ size_t normal_percent = 85;
+ size_t disposable_percent = 10;
+ size_t index_percent = 5;
+ bool has_normal_percent = map.HasMember(CACHE_NORMAL_PERCENT.c_str());
+ bool has_disposable_percent = map.HasMember(CACHE_DISPOSABLE_PERCENT.c_str());
+ bool has_index_percent = map.HasMember(CACHE_INDEX_PERCENT.c_str());
+ if (has_normal_percent && has_disposable_percent && has_index_percent) {
+ RETURN_IF_ERROR(get_percent_value(CACHE_NORMAL_PERCENT, normal_percent));
+ RETURN_IF_ERROR(get_percent_value(CACHE_DISPOSABLE_PERCENT, disposable_percent));
+ RETURN_IF_ERROR(get_percent_value(CACHE_INDEX_PERCENT, index_percent));
+ } else if (has_normal_percent || has_disposable_percent || has_index_percent) {
+ return Status::InvalidArgument(
+ "percent configurations must either be all set or all unset.");
+ }
+ if ((normal_percent + disposable_percent + index_percent) != 100) {
+ return Status::InvalidArgument("the sum is not equal to 100.");
Review Comment:
sum of cache percent config
##########
be/src/olap/options.cpp:
##########
@@ -235,7 +238,38 @@ Status parse_conf_cache_paths(const std::string& config_path, std::vector<CacheP
return Status::InvalidArgument(
"total_size or query_limit should not less than or equal to zero");
}
- paths.emplace_back(std::move(path), total_size, query_limit_bytes);
+
+ // percent
+ auto get_percent_value = [&](const std::string& key, size_t& percent) {
+ auto& value = map.FindMember(key.c_str())->value;
+ if (value.IsUint()) {
+ percent = value.GetUint();
+ } else {
+ return Status::InvalidArgument("percent should be uint");
+ }
+ return Status::OK();
+ };
+
+ size_t normal_percent = 85;
+ size_t disposable_percent = 10;
+ size_t index_percent = 5;
+ bool has_normal_percent = map.HasMember(CACHE_NORMAL_PERCENT.c_str());
+ bool has_disposable_percent = map.HasMember(CACHE_DISPOSABLE_PERCENT.c_str());
+ bool has_index_percent = map.HasMember(CACHE_INDEX_PERCENT.c_str());
+ if (has_normal_percent && has_disposable_percent && has_index_percent) {
+ RETURN_IF_ERROR(get_percent_value(CACHE_NORMAL_PERCENT, normal_percent));
+ RETURN_IF_ERROR(get_percent_value(CACHE_DISPOSABLE_PERCENT, disposable_percent));
+ RETURN_IF_ERROR(get_percent_value(CACHE_INDEX_PERCENT, index_percent));
+ } else if (has_normal_percent || has_disposable_percent || has_index_percent) {
+ return Status::InvalidArgument(
+ "percent configurations must either be all set or all unset.");
Review Comment:
cache percent config
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878336513
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit ac22fd81f29416d6a683e4a835628e5e4c4084ee, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 924 350 334 334
query2 6408 2087 2004 2004
query3 6645 210 201 201
query4 27993 22487 22492 22487
query5 4876 520 559 520
query6 271 201 184 184
query7 4574 279 262 262
query8 231 199 199 199
query9 8218 2529 2467 2467
query10 453 235 236 235
query11 16246 15561 15645 15561
query12 133 76 73 73
query13 1645 322 310 310
query14 11408 7018 7109 7018
query15 225 189 196 189
query16 6398 273 257 257
query17 1849 490 495 490
query18 1919 283 266 266
query19 198 137 136 136
query20 79 77 76 76
query21 183 97 95 95
query22 5106 4626 4657 4626
query23 32075 31204 31182 31182
query24 12122 2795 2757 2757
query25 589 347 335 335
query26 1763 142 141 141
query27 2918 271 271 271
query28 7056 1874 1857 1857
query29 2085 384 396 384
query30 286 143 147 143
query31 985 758 756 756
query32 92 55 58 55
query33 731 254 249 249
query34 913 465 452 452
query35 913 764 770 764
query36 1329 1169 1207 1169
query37 185 68 71 68
query38 3368 3284 3254 3254
query39 1333 1293 1277 1277
query40 309 89 84 84
query41 37 42 35 35
query42 85 86 86 86
query43 529 484 495 484
query44 1057 684 705 684
query45 194 185 180 180
query46 1058 637 642 637
query47 1682 1613 1624 1613
query48 328 254 250 250
query49 1215 313 305 305
query50 776 364 312 312
query51 5460 5276 5271 5271
query52 89 85 93 85
query53 207 150 144 144
query54 1399 578 559 559
query55 84 78 89 78
query56 188 189 195 189
query57 1017 971 923 923
query58 224 199 205 199
query59 2896 2657 2640 2640
query60 222 225 228 225
query61 86 86 84 84
query62 685 458 475 458
query63 169 146 147 146
query64 5829 1687 1715 1687
query65 3337 3268 3257 3257
query66 1338 348 346 346
query67 15925 15585 15465 15465
query68 11141 511 509 509
query69 519 245 255 245
query70 1733 1521 1549 1521
query71 493 224 199 199
query72 5646 3517 3498 3498
query73 2368 309 315 309
query74 6959 6516 6438 6438
query75 4910 2285 2252 2252
query76 6225 1091 1135 1091
query77 663 247 250 247
query78 9149 8641 8747 8641
query79 1036 505 488 488
query80 529 353 339 339
query81 463 204 209 204
query82 209 95 91 91
query83 161 137 133 133
query84 244 53 52 52
query85 915 278 269 269
query86 385 388 387 387
query87 3544 3406 3408 3406
query88 3007 2254 2253 2253
query89 342 259 261 259
query90 1870 194 191 191
query91 120 88 96 88
query92 57 55 49 49
query93 2274 436 482 436
query94 819 178 176 176
query95 458 409 401 401
query96 612 322 320 320
query97 4286 4167 4172 4167
query98 228 196 185 185
query99 1141 840 826 826
Total cold run time: 293656 ms
Total hot run time: 179200 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878766185
TeamCity be ut coverage result:
Function Coverage: 36.61% (8618/23539)
Line Coverage: 28.65% (70090/244654)
Region Coverage: 27.62% (36269/131321)
Branch Coverage: 24.33% (18533/76182)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b57209b85901d160626a14e95ac8310692dbb07c_b57209b85901d160626a14e95ac8310692dbb07c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878419895
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 26.8 seconds inserted 10000000 Rows, about 373K ops/s
storage size: 17183659846 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877043447
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878339300
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.41 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.6 seconds inserted 10000000 Rows, about 362K ops/s
storage size: 17183578581 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878384473
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878738682
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit b57209b85901d160626a14e95ac8310692dbb07c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5476 5117 5167 5117
q2 383 161 158 158
q3 1444 1181 1178 1178
q4 1095 809 855 809
q5 3091 3123 3132 3123
q6 223 134 135 134
q7 979 523 542 523
q8 2148 2285 2231 2231
q9 6692 6688 6626 6626
q10 3165 3180 3121 3121
q11 326 217 225 217
q12 381 234 226 226
q13 4433 3655 3626 3626
q14 246 213 215 213
q15 584 550 518 518
q16 447 386 402 386
q17 1049 475 527 475
q18 7118 6707 7076 6707
q19 1624 1452 1596 1452
q20 589 425 370 370
q21 2891 2436 2510 2436
q22 412 343 331 331
Total cold run time: 44796 ms
Total hot run time: 39977 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5167 5100 5169 5100
q2 344 229 253 229
q3 3352 3343 3335 3335
q4 2153 2044 2036 2036
q5 5961 5920 5952 5920
q6 222 126 130 126
q7 2369 1896 1925 1896
q8 3557 3658 3655 3655
q9 9066 8972 9048 8972
q10 3854 3896 3891 3891
q11 593 493 493 493
q12 808 640 631 631
q13 3874 3194 3236 3194
q14 295 265 284 265
q15 602 534 531 531
q16 551 494 516 494
q17 2053 1803 1805 1803
q18 8842 8426 8403 8403
q19 1748 1663 1700 1663
q20 2279 1962 1974 1962
q21 5651 5308 5364 5308
q22 542 505 508 505
Total cold run time: 63883 ms
Total hot run time: 60412 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1879525384
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1879531999
TeamCity be ut coverage result:
Function Coverage: 36.62% (8621/23540)
Line Coverage: 28.66% (70127/244670)
Region Coverage: 27.63% (36288/131328)
Branch Coverage: 24.34% (18540/76186)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ffb35884eec079859d839faac283980d115e659b_ffb35884eec079859d839faac283980d115e659b/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878758608
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.14 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 27.4 seconds inserted 10000000 Rows, about 364K ops/s
storage size: 17184293395 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1876898699
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 1e38fef227a5186bb4dbb63fe11b97b5e27ab93b, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5429 5114 5203 5114
q2 394 161 166 161
q3 1461 1140 1067 1067
q4 1073 837 800 800
q5 3102 3163 3152 3152
q6 224 136 134 134
q7 983 555 510 510
q8 2137 2281 2241 2241
q9 6704 6709 6643 6643
q10 3175 3106 3141 3106
q11 348 226 214 214
q12 383 233 237 233
q13 4439 3654 3635 3635
q14 249 212 221 212
q15 571 541 533 533
q16 463 409 416 409
q17 1055 554 564 554
q18 7099 6727 6775 6727
q19 1642 1501 1437 1437
q20 591 352 340 340
q21 2926 2479 2477 2477
q22 399 332 330 330
Total cold run time: 44847 ms
Total hot run time: 40029 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5112 5008 5051 5008
q2 344 248 243 243
q3 3388 3362 3317 3317
q4 2152 2087 2056 2056
q5 5974 5932 5938 5932
q6 225 126 130 126
q7 2401 1929 1905 1905
q8 3579 3644 3687 3644
q9 9046 8972 8966 8966
q10 3895 3902 3910 3902
q11 582 471 484 471
q12 791 650 654 650
q13 3878 3189 3177 3177
q14 303 271 277 271
q15 583 523 525 523
q16 552 513 483 483
q17 2019 1835 1798 1798
q18 8741 8315 8296 8296
q19 1739 1680 1686 1680
q20 2258 1961 1955 1955
q21 5792 5361 5284 5284
q22 576 515 509 509
Total cold run time: 63930 ms
Total hot run time: 60196 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877319403
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.3 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17188959192 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878105583
TeamCity be ut coverage result:
Function Coverage: 36.64% (8617/23521)
Line Coverage: 28.67% (70044/244276)
Region Coverage: 27.65% (36259/131130)
Branch Coverage: 24.35% (18525/76082)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8d19aa883733a3766de71ea2b13f716b1018a658_8d19aa883733a3766de71ea2b13f716b1018a658/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878265371
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878432735
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit e6da27ac2fe876f645d95825181674f36ffd7491, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 891 348 337 337
query2 4934 1934 1903 1903
query3 5490 211 206 206
query4 26644 22596 22512 22512
query5 3386 495 538 495
query6 258 184 181 181
query7 4202 274 256 256
query8 224 201 189 189
query9 7618 2570 2556 2556
query10 404 255 248 248
query11 16224 15563 15558 15558
query12 122 75 77 75
query13 1530 323 317 317
query14 10186 7160 7142 7142
query15 210 185 190 185
query16 6082 271 262 262
query17 1608 495 487 487
query18 1788 265 270 265
query19 167 137 130 130
query20 78 78 77 77
query21 169 100 93 93
query22 5080 4866 4724 4724
query23 32277 31291 31121 31121
query24 11956 2811 2785 2785
query25 598 333 332 332
query26 1730 139 136 136
query27 2866 279 269 269
query28 6777 1879 1874 1874
query29 2146 384 379 379
query30 289 140 146 140
query31 1011 777 776 776
query32 84 56 56 56
query33 722 260 250 250
query34 883 446 438 438
query35 890 774 751 751
query36 1248 1192 1222 1192
query37 152 68 72 68
query38 3396 3316 3280 3280
query39 1324 1289 1267 1267
query40 285 90 84 84
query41 38 36 35 35
query42 93 91 84 84
query43 515 528 475 475
query44 1072 690 702 690
query45 198 188 179 179
query46 1057 633 633 633
query47 1623 1613 1545 1545
query48 335 253 247 247
query49 1079 308 316 308
query50 806 324 326 324
query51 5347 5333 5261 5261
query52 93 86 77 77
query53 210 144 139 139
query54 1486 561 559 559
query55 94 82 87 82
query56 193 190 194 190
query57 1025 963 945 945
query58 218 195 209 195
query59 2796 2544 2691 2544
query60 243 215 228 215
query61 83 85 84 84
query62 670 437 451 437
query63 163 153 147 147
query64 5354 1744 1748 1744
query65 3334 3295 3286 3286
query66 1346 344 330 330
query67 15859 15210 15526 15210
query68 11249 513 542 513
query69 479 247 259 247
query70 1559 1453 1503 1453
query71 474 209 205 205
query72 5580 3595 3561 3561
query73 2263 314 308 308
query74 7043 6509 6470 6470
query75 4754 2299 2337 2299
query76 5935 1059 1119 1059
query77 680 237 258 237
query78 9163 8600 8561 8561
query79 2090 492 492 492
query80 689 347 340 340
query81 461 209 208 208
query82 208 91 92 91
query83 166 139 133 133
query84 242 55 53 53
query85 973 273 277 273
query86 382 377 386 377
query87 3542 3334 3373 3334
query88 3124 2238 2240 2238
query89 360 250 267 250
query90 1994 197 187 187
query91 124 90 90 90
query92 61 52 54 52
query93 2909 481 483 481
query94 843 181 174 174
query95 450 426 397 397
query96 627 321 317 317
query97 4263 4157 4155 4155
query98 208 202 187 187
query99 1105 813 850 813
Total cold run time: 285259 ms
Total hot run time: 178916 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878417688
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit e6da27ac2fe876f645d95825181674f36ffd7491, data reload: false
------ Round 1 ----------------------------------
q1 17887 5590 5087 5087
q2 2023 156 143 143
q3 10623 1128 1169 1128
q4 10237 791 822 791
q5 7784 2970 3005 2970
q6 212 133 132 132
q7 917 556 512 512
q8 9238 2026 1996 1996
q9 6846 6452 6446 6446
q10 8243 3022 3198 3022
q11 421 211 211 211
q12 385 229 227 227
q13 18009 3614 3608 3608
q14 247 212 220 212
q15 560 527 519 519
q16 449 410 396 396
q17 977 466 513 466
q18 7560 6751 6725 6725
q19 1578 1382 1338 1338
q20 731 344 336 336
q21 2813 2383 2418 2383
q22 377 330 315 315
Total cold run time: 108117 ms
Total hot run time: 38963 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5189 5143 5165 5143
q2 349 225 240 225
q3 3320 3337 3292 3292
q4 2150 2019 2050 2019
q5 5847 5796 5794 5794
q6 219 126 127 126
q7 2333 1947 1943 1943
q8 3390 3446 3468 3446
q9 8846 8809 8815 8809
q10 3802 3846 3831 3831
q11 582 504 490 490
q12 794 627 641 627
q13 9754 3240 3231 3231
q14 287 270 278 270
q15 583 519 517 517
q16 547 488 515 488
q17 1970 1760 1769 1760
q18 8731 8444 8390 8390
q19 1629 1800 1554 1554
q20 2192 1978 1954 1954
q21 5612 5326 5245 5245
q22 529 472 506 472
Total cold run time: 68655 ms
Total hot run time: 59626 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878951749
TeamCity be ut coverage result:
Function Coverage: 36.61% (8618/23540)
Line Coverage: 28.65% (70101/244670)
Region Coverage: 27.62% (36274/131328)
Branch Coverage: 24.33% (18537/76186)
Coverage Report: http://coverage.selectdb-in.cc/coverage/195d434ae08b6857e698e31ae9a597ae3c7364c4_195d434ae08b6857e698e31ae9a597ae3c7364c4/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](file cache) make the cache queue percent configurable [doris]
Posted by "qidaye (via GitHub)" <gi...@apache.org>.
qidaye merged PR #29537:
URL: https://github.com/apache/doris/pull/29537
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878111410
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.89 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 27.1 seconds inserted 10000000 Rows, about 369K ops/s
storage size: 17183383929 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878104381
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 8d19aa883733a3766de71ea2b13f716b1018a658, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5481 5151 5181 5151
q2 385 189 144 144
q3 1463 1170 1170 1170
q4 1100 843 742 742
q5 3130 3115 3092 3092
q6 222 132 136 132
q7 968 551 506 506
q8 2162 2283 2216 2216
q9 6716 6683 6706 6683
q10 3181 3151 3130 3130
q11 331 224 222 222
q12 377 246 237 237
q13 4438 3672 3625 3625
q14 257 230 212 212
q15 577 533 534 533
q16 448 400 402 400
q17 1059 515 511 511
q18 7069 6708 6739 6708
q19 1652 1482 1410 1410
q20 585 330 336 330
q21 2899 2510 2582 2510
q22 397 318 323 318
Total cold run time: 44897 ms
Total hot run time: 39982 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5151 5084 5046 5046
q2 335 229 240 229
q3 3377 3349 3347 3347
q4 2159 2030 2045 2030
q5 5967 5933 5977 5933
q6 227 125 129 125
q7 2383 1952 1946 1946
q8 3575 3649 3683 3649
q9 9009 9042 8991 8991
q10 3866 3889 3907 3889
q11 565 481 476 476
q12 819 652 652 652
q13 3886 3200 3216 3200
q14 301 266 267 266
q15 583 521 531 521
q16 563 498 501 498
q17 2029 1804 1813 1804
q18 8813 8396 8315 8315
q19 1744 1695 1655 1655
q20 2275 1989 1962 1962
q21 5742 5463 5345 5345
q22 569 505 474 474
Total cold run time: 63938 ms
Total hot run time: 60353 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878755190
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit b57209b85901d160626a14e95ac8310692dbb07c, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 933 345 333 333
query2 6412 1870 1960 1870
query3 6661 204 196 196
query4 26675 22506 22191 22191
query5 3877 501 491 491
query6 264 182 175 175
query7 4576 271 258 258
query8 224 196 198 196
query9 8306 2554 2556 2554
query10 428 245 238 238
query11 16109 15715 15528 15528
query12 139 74 75 74
query13 1635 335 318 318
query14 11578 6971 7106 6971
query15 220 185 190 185
query16 6308 273 255 255
query17 1881 490 479 479
query18 1946 278 263 263
query19 180 129 134 129
query20 82 72 76 72
query21 186 102 97 97
query22 4705 4567 4315 4315
query23 32278 31271 31479 31271
query24 12179 2840 2837 2837
query25 597 335 332 332
query26 1797 135 137 135
query27 2903 268 279 268
query28 7072 1907 1888 1888
query29 2138 392 391 391
query30 295 141 145 141
query31 1005 766 770 766
query32 89 56 53 53
query33 747 268 250 250
query34 932 441 442 441
query35 866 772 803 772
query36 1268 1197 1233 1197
query37 187 70 69 69
query38 3437 3287 3283 3283
query39 1326 1290 1276 1276
query40 296 85 94 85
query41 39 34 34 34
query42 91 80 84 80
query43 532 479 487 479
query44 1057 693 705 693
query45 193 183 183 183
query46 1071 634 647 634
query47 1670 1482 1508 1482
query48 328 255 262 255
query49 1201 312 306 306
query50 725 316 345 316
query51 5385 5263 5280 5263
query52 100 84 93 84
query53 213 146 145 145
query54 1379 556 558 556
query55 95 86 74 74
query56 186 186 190 186
query57 1024 901 970 901
query58 234 189 189 189
query59 2841 2573 2610 2573
query60 222 228 225 225
query61 82 81 80 80
query62 636 431 450 431
query63 163 144 144 144
query64 5867 1710 1677 1677
query65 3314 3250 3250 3250
query66 1385 341 317 317
query67 15602 15123 15334 15123
query68 14008 507 517 507
query69 524 245 253 245
query70 2241 1516 1531 1516
query71 487 206 196 196
query72 5611 3509 3481 3481
query73 3553 317 308 308
query74 7063 6495 6483 6483
query75 5414 2260 2292 2260
query76 6274 1096 1126 1096
query77 709 239 252 239
query78 9180 8558 8697 8558
query79 2751 497 498 497
query80 588 347 333 333
query81 462 206 205 205
query82 206 95 93 93
query83 175 138 134 134
query84 245 53 51 51
query85 969 277 261 261
query86 436 423 382 382
query87 3546 3369 3413 3369
query88 3023 2229 2209 2209
query89 325 240 249 240
query90 2013 191 187 187
query91 115 93 89 89
query92 56 46 51 46
query93 1639 500 500 500
query94 883 178 175 175
query95 461 413 400 400
query96 615 312 312 312
query97 4263 4142 4137 4137
query98 203 193 188 188
query99 1102 755 843 755
Total cold run time: 297512 ms
Total hot run time: 177775 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877039024
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877052739
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877111585
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.42 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183423956 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877296391
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit acbb64ff690dc4f827a4c8dc8765253a86d3070c, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5506 5157 5168 5157
q2 381 175 177 175
q3 1461 1167 1110 1110
q4 1084 874 787 787
q5 3145 3095 3053 3053
q6 230 131 130 130
q7 970 551 515 515
q8 2159 2296 2229 2229
q9 6691 6685 6683 6683
q10 3185 3126 3157 3126
q11 345 211 218 211
q12 379 238 242 238
q13 4400 3627 3661 3627
q14 244 232 213 213
q15 587 537 535 535
q16 467 403 384 384
q17 1041 482 592 482
q18 7096 6705 6976 6705
q19 1652 1522 1583 1522
q20 585 342 350 342
q21 2963 2508 2482 2482
q22 409 333 328 328
Total cold run time: 44980 ms
Total hot run time: 40034 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5160 5086 5071 5071
q2 336 239 237 237
q3 3342 3341 3336 3336
q4 2178 2006 1988 1988
q5 5936 5924 5920 5920
q6 222 125 128 125
q7 2397 1930 1935 1930
q8 3555 3651 3685 3651
q9 9019 8962 9002 8962
q10 3875 3886 3925 3886
q11 578 501 467 467
q12 799 636 635 635
q13 3916 3209 3222 3209
q14 307 284 274 274
q15 585 539 523 523
q16 578 483 491 483
q17 2047 1826 1798 1798
q18 8744 8365 8512 8365
q19 1747 1711 1709 1709
q20 2272 1979 1967 1967
q21 5810 5416 5286 5286
q22 547 534 539 534
Total cold run time: 63950 ms
Total hot run time: 60356 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877293823
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit acbb64ff690dc4f827a4c8dc8765253a86d3070c, data reload: false
------ Round 1 ----------------------------------
q1 17677 5205 5129 5129
q2 2023 154 140 140
q3 10555 1069 1113 1069
q4 10174 786 826 786
q5 7751 2924 2904 2904
q6 208 134 170 134
q7 912 525 497 497
q8 9291 2013 2017 2013
q9 6830 6428 6392 6392
q10 8227 3041 3005 3005
q11 435 220 221 220
q12 390 231 234 231
q13 18023 3625 3662 3625
q14 233 219 221 219
q15 558 520 530 520
q16 439 389 384 384
q17 960 506 595 506
q18 7339 6796 6707 6707
q19 1566 1428 1345 1345
q20 697 349 316 316
q21 2789 2312 2413 2312
q22 388 348 338 338
Total cold run time: 107465 ms
Total hot run time: 38792 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5158 5103 5114 5103
q2 344 229 245 229
q3 3294 3271 3291 3271
q4 2112 2002 2038 2002
q5 5824 5805 5786 5786
q6 219 125 126 125
q7 2334 1941 1907 1907
q8 3382 3428 3476 3428
q9 8933 8764 8736 8736
q10 3804 3853 3881 3853
q11 591 489 495 489
q12 784 627 632 627
q13 8105 3245 3335 3245
q14 307 297 283 283
q15 582 506 526 506
q16 573 519 534 519
q17 1979 1792 1808 1792
q18 8732 8364 8247 8247
q19 1624 1597 1599 1597
q20 2206 1990 1962 1962
q21 5555 5331 5243 5243
q22 536 519 491 491
Total cold run time: 66978 ms
Total hot run time: 59441 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878079060
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877008233
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877030009
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877217352
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878264009
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878109853
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 8d19aa883733a3766de71ea2b13f716b1018a658, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 928 344 336 336
query2 6427 1851 1923 1851
query3 6651 211 203 203
query4 26690 22457 22328 22328
query5 3879 546 496 496
query6 266 179 182 179
query7 4587 260 254 254
query8 232 212 199 199
query9 8168 2520 2493 2493
query10 411 232 242 232
query11 16189 15551 15680 15551
query12 135 73 72 72
query13 1679 317 316 316
query14 12193 7172 6995 6995
query15 217 183 197 183
query16 6406 270 259 259
query17 1864 489 480 480
query18 1933 260 258 258
query19 189 135 131 131
query20 79 73 76 73
query21 181 96 94 94
query22 5019 4783 4538 4538
query23 31931 31037 31086 31037
query24 11853 2778 2813 2778
query25 598 341 338 338
query26 1718 137 141 137
query27 2885 271 267 267
query28 6989 1871 1853 1853
query29 2080 380 393 380
query30 287 140 146 140
query31 990 757 765 757
query32 92 53 48 48
query33 733 246 247 246
query34 931 432 442 432
query35 899 777 774 774
query36 1332 1226 1149 1149
query37 181 63 64 63
query38 3369 3265 3227 3227
query39 1304 1285 1280 1280
query40 301 90 86 86
query41 37 36 40 36
query42 89 81 84 81
query43 525 492 488 488
query44 1128 687 693 687
query45 205 190 181 181
query46 1060 629 642 629
query47 1718 1585 1493 1493
query48 332 250 252 250
query49 1217 303 312 303
query50 735 366 351 351
query51 5383 5270 5244 5244
query52 96 82 85 82
query53 207 151 142 142
query54 1408 553 566 553
query55 88 83 88 83
query56 187 183 189 183
query57 1022 946 878 878
query58 213 192 201 192
query59 2782 2633 2639 2633
query60 223 210 217 210
query61 86 85 86 85
query62 646 464 464 464
query63 157 146 140 140
query64 5868 1745 1696 1696
query65 3303 3231 3252 3231
query66 1389 330 339 330
query67 15661 15181 15297 15181
query68 12037 519 503 503
query69 501 245 245 245
query70 1699 1596 1476 1476
query71 491 226 219 219
query72 5716 3662 3624 3624
query73 2703 314 302 302
query74 7023 6453 6409 6409
query75 4914 2263 2273 2263
query76 6288 1149 1093 1093
query77 671 258 280 258
query78 9103 8568 8617 8568
query79 1053 487 486 486
query80 578 343 334 334
query81 469 206 208 206
query82 205 85 98 85
query83 193 133 132 132
query84 246 52 54 52
query85 981 284 265 265
query86 389 393 397 393
query87 3559 3361 3346 3346
query88 2969 2214 2217 2214
query89 318 251 249 249
query90 1986 188 187 187
query91 120 89 88 88
query92 56 49 50 49
query93 1184 472 477 472
query94 920 171 171 171
query95 436 399 405 399
query96 617 311 315 311
query97 4281 4135 4117 4117
query98 207 189 192 189
query99 1082 794 831 794
Total cold run time: 291556 ms
Total hot run time: 177838 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878634317
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878348367
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit ac22fd81f29416d6a683e4a835628e5e4c4084ee, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5454 5111 5117 5111
q2 381 173 158 158
q3 1453 1136 1188 1136
q4 1077 854 856 854
q5 3148 3117 3122 3117
q6 220 131 131 131
q7 953 551 511 511
q8 2144 2257 2276 2257
q9 6729 6683 6637 6637
q10 3170 3118 3131 3118
q11 353 219 217 217
q12 386 233 234 233
q13 4405 3635 3644 3635
q14 250 215 221 215
q15 591 542 522 522
q16 457 390 391 390
q17 1044 532 537 532
q18 7108 6745 6771 6745
q19 1628 1484 1426 1426
q20 572 318 333 318
q21 2865 2368 2471 2368
q22 373 330 339 330
Total cold run time: 44761 ms
Total hot run time: 39961 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5164 4989 5054 4989
q2 338 235 241 235
q3 3367 3342 3315 3315
q4 2154 2042 2038 2038
q5 6013 5964 5925 5925
q6 223 124 128 124
q7 2386 1930 1900 1900
q8 3580 3647 3663 3647
q9 9037 8937 8969 8937
q10 3863 3873 3905 3873
q11 585 469 491 469
q12 795 637 639 637
q13 3879 3206 3180 3180
q14 302 278 277 277
q15 584 524 528 524
q16 552 509 495 495
q17 2015 1801 1834 1801
q18 8734 8443 8455 8443
q19 1734 1659 1697 1659
q20 2271 1988 1978 1978
q21 5790 5318 5327 5318
q22 550 498 492 492
Total cold run time: 63916 ms
Total hot run time: 60256 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1879531208
<details>
<summary>TPC-H: <b>Total hot run time: 38316 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ffb35884eec079859d839faac283980d115e659b, data reload: false
------ Round 1 ----------------------------------
q1 17654 4938 4922 4922
q2 2004 149 138 138
q3 10600 1088 1100 1088
q4 10207 773 815 773
q5 7780 2940 2899 2899
q6 200 121 119 119
q7 941 524 484 484
q8 9233 1975 1967 1967
q9 6463 6347 6331 6331
q10 8219 3112 2996 2996
q11 448 208 220 208
q12 387 235 228 228
q13 18019 3389 3373 3373
q14 238 208 205 205
q15 538 510 498 498
q16 448 376 405 376
q17 955 709 632 632
q18 7301 6602 6631 6602
q19 1568 1507 1516 1507
q20 712 328 326 326
q21 2679 2418 2331 2331
q22 363 313 315 313
Total cold run time: 106957 ms
Total hot run time: 38316 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4899 4940 4932 4932
q2 342 243 236 236
q3 3286 3282 3288 3282
q4 2094 2024 2016 2016
q5 5758 5787 5766 5766
q6 203 119 115 115
q7 2269 1882 1893 1882
q8 3390 3436 3442 3436
q9 8743 8761 8726 8726
q10 3757 3827 3832 3827
q11 539 452 457 452
q12 810 632 633 632
q13 7469 3202 3168 3168
q14 297 258 259 258
q15 547 506 507 506
q16 569 504 482 482
q17 1854 1853 1870 1853
q18 8685 8341 8211 8211
q19 1592 1634 1622 1622
q20 2190 1965 1941 1941
q21 5563 5216 5227 5216
q22 502 466 458 458
Total cold run time: 65358 ms
Total hot run time: 59017 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1879526409
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1876790371
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1876898546
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.96 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17183951108 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877107022
TPC-DS test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpcds-tools
```
TPC-DS sf100 test result on commit 738cea3bf39cd23b4ef8505cb42c393fbefffb82, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 904 348 332 332
query2 6431 2044 1914 1914
query3 6639 210 202 202
query4 27165 22493 22489 22489
query5 4653 514 493 493
query6 268 179 169 169
query7 4582 258 262 258
query8 226 201 191 191
query9 8189 2506 2514 2506
query10 417 241 246 241
query11 16179 15550 15527 15527
query12 138 77 74 74
query13 1649 316 314 314
query14 11798 7096 7103 7096
query15 216 178 191 178
query16 6440 267 262 262
query17 1837 487 488 487
query18 1924 265 264 264
query19 181 134 137 134
query20 79 73 73 73
query21 181 96 89 89
query22 5119 4782 4598 4598
query23 31914 31225 31129 31129
query24 11929 2782 2801 2782
query25 595 332 340 332
query26 1772 138 139 138
query27 2917 269 265 265
query28 7024 1868 1878 1868
query29 2085 376 377 376
query30 285 139 146 139
query31 981 764 776 764
query32 87 56 56 56
query33 736 253 262 253
query34 900 438 445 438
query35 909 748 787 748
query36 1319 1224 1203 1203
query37 181 66 68 66
query38 3420 3300 3315 3300
query39 1316 1285 1273 1273
query40 299 85 86 85
query41 37 34 32 32
query42 87 82 88 82
query43 525 537 497 497
query44 1033 684 698 684
query45 197 187 182 182
query46 1074 640 655 640
query47 1652 1632 1531 1531
query48 319 252 266 252
query49 1220 314 308 308
query50 745 316 319 316
query51 5455 5250 5354 5250
query52 92 86 87 86
query53 205 146 142 142
query54 1371 559 551 551
query55 91 87 86 86
query56 188 188 186 186
query57 1016 956 988 956
query58 212 196 195 195
query59 2808 2528 2607 2528
query60 222 214 215 214
query61 87 86 89 86
query62 638 453 466 453
query63 166 146 148 146
query64 5922 1751 1685 1685
query65 3343 3248 3234 3234
query66 1364 341 336 336
query67 15729 15683 15238 15238
query68 10603 510 529 510
query69 515 268 263 263
query70 1763 1584 1620 1584
query71 492 224 203 203
query72 5428 3625 3582 3582
query73 2161 306 303 303
query74 7013 6524 6484 6484
query75 4806 2288 2299 2288
query76 6310 1112 1122 1112
query77 657 241 267 241
query78 9073 8774 8549 8549
query79 1024 497 490 490
query80 529 345 335 335
query81 454 209 204 204
query82 208 89 89 89
query83 161 135 132 132
query84 244 52 52 52
query85 938 281 271 271
query86 387 379 390 379
query87 3553 3389 3315 3315
query88 3600 2245 2231 2231
query89 329 251 258 251
query90 1868 191 194 191
query91 115 88 86 86
query92 59 50 50 50
query93 1872 418 406 406
query94 900 183 177 177
query95 463 411 400 400
query96 637 320 323 320
query97 4299 4179 4148 4148
query98 209 195 192 192
query99 1104 863 845 845
Total cold run time: 291456 ms
Total hot run time: 178685 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877104822
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 738cea3bf39cd23b4ef8505cb42c393fbefffb82, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5503 5239 5196 5196
q2 389 166 158 158
q3 1461 1152 1174 1152
q4 1093 763 728 728
q5 3089 3140 3032 3032
q6 223 133 133 133
q7 966 544 540 540
q8 2154 2219 2281 2219
q9 6716 6651 6636 6636
q10 3197 3142 3176 3142
q11 350 235 215 215
q12 385 236 232 232
q13 4428 3666 3606 3606
q14 250 227 223 223
q15 585 527 523 523
q16 449 402 397 397
q17 1048 534 510 510
q18 7102 6743 6794 6743
q19 1641 1534 1412 1412
q20 602 330 323 323
q21 2894 2559 2467 2467
q22 395 343 356 343
Total cold run time: 44920 ms
Total hot run time: 39930 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5062 5071 5083 5071
q2 342 234 244 234
q3 3355 3340 3306 3306
q4 2150 2063 2066 2063
q5 5935 5969 5930 5930
q6 222 126 124 124
q7 2415 1905 1962 1905
q8 3577 3640 3684 3640
q9 9066 8951 9002 8951
q10 3887 3924 3896 3896
q11 608 492 497 492
q12 797 633 639 633
q13 3900 3180 3169 3169
q14 310 282 275 275
q15 594 529 531 529
q16 551 536 500 500
q17 2013 1828 1823 1823
q18 8730 8416 8347 8347
q19 1736 1726 1706 1706
q20 2296 1989 1984 1984
q21 5825 5283 5285 5283
q22 588 508 499 499
Total cold run time: 63959 ms
Total hot run time: 60360 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percentage configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1876878474
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 1e38fef227a5186bb4dbb63fe11b97b5e27ab93b, data reload: false
------ Round 1 ----------------------------------
q1 18439 5272 5085 5085
q2 2024 152 141 141
q3 10627 1085 1168 1085
q4 10236 794 796 794
q5 7815 3076 2995 2995
q6 234 140 140 140
q7 939 502 551 502
q8 9285 2037 2059 2037
q9 7364 6413 6463 6413
q10 8269 3082 3063 3063
q11 444 220 235 220
q12 395 238 230 230
q13 18013 3659 3687 3659
q14 249 216 216 216
q15 568 527 521 521
q16 446 395 399 395
q17 981 508 517 508
q18 7345 6736 6663 6663
q19 1627 1375 1416 1375
q20 731 350 349 349
q21 2815 2386 2433 2386
q22 381 308 318 308
Total cold run time: 109227 ms
Total hot run time: 39085 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5169 5120 5052 5052
q2 336 233 245 233
q3 3337 3286 3298 3286
q4 2148 2014 2014 2014
q5 6040 5803 5814 5803
q6 215 126 131 126
q7 2328 1924 1924 1924
q8 3395 3503 3524 3503
q9 8859 8821 8855 8821
q10 3808 3853 3820 3820
q11 571 473 495 473
q12 777 625 643 625
q13 7290 3221 3238 3221
q14 295 260 280 260
q15 579 526 528 526
q16 551 496 503 496
q17 1961 1765 1781 1765
q18 8738 8393 8295 8295
q19 1635 1601 1617 1601
q20 2183 1986 1935 1935
q21 5635 5308 5276 5276
q22 548 493 493 493
Total cold run time: 66398 ms
Total hot run time: 59548 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1877309758
TeamCity be ut coverage result:
Function Coverage: 36.64% (8617/23517)
Line Coverage: 28.69% (70052/244201)
Region Coverage: 27.66% (36259/131085)
Branch Coverage: 24.35% (18518/76050)
Coverage Report: http://coverage.selectdb-in.cc/coverage/acbb64ff690dc4f827a4c8dc8765253a86d3070c_acbb64ff690dc4f827a4c8dc8765253a86d3070c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878075340
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878263306
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878374979
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1879542107
<details>
<summary>TPC-DS: <b>Total hot run time: 183667 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ffb35884eec079859d839faac283980d115e659b, data reload: false
run tpcds-sf100 query with default conf and session variables
query1 934 337 332 332
query2 6566 2035 2009 2009
query3 6722 229 220 220
query4 26914 22530 22322 22322
query5 6122 602 618 602
query6 341 207 195 195
query7 4629 299 304 299
query8 240 232 224 224
query9 8638 2889 2862 2862
query10 614 319 315 315
query11 16226 15638 15587 15587
query12 167 99 97 97
query13 1666 337 344 337
query14 13136 7581 7471 7471
query15 284 223 210 210
query16 6369 333 324 324
query17 1711 512 508 508
query18 1973 304 295 295
query19 320 157 155 155
query20 98 101 96 96
query21 183 93 102 93
query22 5018 4933 4914 4914
query23 31931 31378 31519 31378
query24 11833 2932 2881 2881
query25 607 366 367 366
query26 1791 156 165 156
query27 2873 317 334 317
query28 7196 2118 2113 2113
query29 2037 413 411 411
query30 288 143 146 143
query31 1028 814 816 814
query32 135 85 73 73
query33 839 333 356 333
query34 871 463 469 463
query35 965 861 843 843
query36 1435 1229 1257 1229
query37 208 94 81 81
query38 3570 3417 3344 3344
query39 1360 1303 1279 1279
query40 301 90 85 85
query41 38 41 35 35
query42 107 101 103 101
query43 544 516 533 516
query44 1348 812 816 812
query45 212 202 204 202
query46 1029 687 722 687
query47 1742 1656 1629 1629
query48 360 282 290 282
query49 1222 317 319 317
query50 732 314 347 314
query51 5349 5384 5336 5336
query52 108 97 94 94
query53 235 159 163 159
query54 1447 670 676 670
query55 105 105 99 99
query56 274 241 275 241
query57 1130 973 969 969
query58 294 276 256 256
query59 3001 2845 2818 2818
query60 317 296 296 296
query61 154 147 149 147
query62 571 474 499 474
query63 180 176 166 166
query64 5950 1741 1728 1728
query65 3405 3321 3319 3319
query66 1309 372 386 372
query67 15703 15294 15300 15294
query68 13684 546 547 546
query69 616 318 298 298
query70 1791 1599 1645 1599
query71 596 295 283 283
query72 5340 3437 3453 3437
query73 3276 348 337 337
query74 7060 6526 6458 6458
query75 5324 2287 2271 2271
query76 6439 1087 1183 1087
query77 810 328 306 306
query78 9154 8478 8477 8477
query79 1888 517 524 517
query80 549 341 353 341
query81 463 205 218 205
query82 228 102 102 102
query83 193 143 141 141
query84 257 56 58 56
query85 942 268 267 267
query86 435 445 428 428
query87 3597 3459 3524 3459
query88 3238 2456 2494 2456
query89 365 284 278 278
query90 1931 268 252 252
query91 131 95 98 95
query92 87 89 69 69
query93 1160 435 505 435
query94 923 244 240 240
query95 582 502 496 496
query96 631 327 325 325
query97 4347 4238 4242 4238
query98 242 214 211 211
query99 1133 859 848 848
Total cold run time: 302417 ms
Total hot run time: 183667 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on code in PR #29537:
URL: https://github.com/apache/doris/pull/29537#discussion_r1442467448
##########
be/src/olap/options.cpp:
##########
@@ -244,18 +268,18 @@ Status parse_conf_cache_paths(const std::string& config_path, std::vector<CacheP
return Status::OK();
}
-io::FileCacheSettings CachePath::init_settings() const {
+io::FileCacheSettings CachePath::init_settings(const std::vector<size_t>& percents) const {
Review Comment:
The percents is member variable. Not need the input param.
##########
be/test/io/cache/file_block_cache_test.cpp:
##########
@@ -118,7 +118,7 @@ TEST(LRUFileCache, init) {
EXPECT_TRUE(parse_conf_cache_paths(string, cache_paths));
EXPECT_EQ(cache_paths.size(), 2);
for (const auto& cache_path : cache_paths) {
- io::FileCacheSettings settings = cache_path.init_settings();
+ io::FileCacheSettings settings = cache_path.init_settings(cache_path.percents);
Review Comment:
Add some ut to cover it.
##########
be/src/olap/options.cpp:
##########
@@ -235,7 +236,30 @@ Status parse_conf_cache_paths(const std::string& config_path, std::vector<CacheP
return Status::InvalidArgument(
"total_size or query_limit should not less than or equal to zero");
}
- paths.emplace_back(std::move(path), total_size, query_limit_bytes);
+
+ // percent
+ std::vector<size_t> percents;
+ if (map.HasMember(CACHE_PERCENTS.c_str())) {
+ auto& percentArray = map.FindMember(CACHE_PERCENTS.c_str())->value;
+ if (percentArray.IsArray()) {
+ for (auto& v : percentArray.GetArray()) {
+ if (v.IsUint()) {
+ percents.push_back(v.GetUint());
+ } else {
+ return Status::InvalidArgument("percent should be uint");
+ }
+ }
+ } else {
+ return Status::InvalidArgument("percents should be array");
+ }
+ } else {
+ percents.assign({85, 10, 5});
+ }
+ if (percents.size() != 3 || percents[0] + percents[1] + percents[2] != 100) {
+ return Status::InvalidArgument("wrong percents value");
Review Comment:
More detail. The array size is not equal to 3 or the sum is not equal to 100.
##########
be/src/olap/options.cpp:
##########
@@ -200,7 +201,7 @@ void parse_conf_broken_store_paths(const string& config_path, std::set<std::stri
* [
* {"path": "storage1", "total_size":53687091200,"query_limit": "10737418240"},
* {"path": "storage2", "total_size":53687091200},
- * {"path": "storage3", "total_size":53687091200},
+ * {"path": "storage3", "total_size":53687091200, percents: [85,10,5]}
Review Comment:
Modify the annotation in "config.h" too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878830567
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1878901178
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools
```
Tpch sf100 test result on commit 195d434ae08b6857e698e31ae9a597ae3c7364c4, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5522 5110 5097 5097
q2 382 161 146 146
q3 1453 1232 1173 1173
q4 1090 828 848 828
q5 3130 3132 3117 3117
q6 222 132 137 132
q7 958 567 536 536
q8 2145 2277 2296 2277
q9 6752 6654 6669 6654
q10 3152 3127 3081 3081
q11 343 216 234 216
q12 377 239 231 231
q13 4389 3631 3632 3631
q14 248 221 211 211
q15 578 534 535 534
q16 450 422 418 418
q17 1054 550 530 530
q18 7083 6801 6726 6726
q19 1649 1508 1398 1398
q20 606 341 321 321
q21 2878 2489 2474 2474
q22 403 319 335 319
Total cold run time: 44864 ms
Total hot run time: 40050 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5079 5077 5068 5068
q2 329 235 231 231
q3 3361 3362 3311 3311
q4 2156 2013 2013 2013
q5 6010 5892 5957 5892
q6 222 127 130 127
q7 2376 1939 1921 1921
q8 3579 3648 3673 3648
q9 9072 9014 8983 8983
q10 3861 3892 3895 3892
q11 597 496 481 481
q12 799 633 641 633
q13 3867 3199 3227 3199
q14 301 268 284 268
q15 590 516 523 516
q16 544 512 475 475
q17 2035 1837 1808 1808
q18 8732 8370 8301 8301
q19 1740 1715 1685 1685
q20 2273 1982 1969 1969
q21 5762 5293 5306 5293
q22 548 513 476 476
Total cold run time: 63833 ms
Total hot run time: 60190 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] (opt)(file cache) make the cache queue percent configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #29537:
URL: https://github.com/apache/doris/pull/29537#issuecomment-1879545417
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.5 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 26.4 seconds inserted 10000000 Rows, about 378K ops/s
storage size: 17183829826 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org