You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Jibing-Li (via GitHub)" <gi...@apache.org> on 2024/02/27 08:03:50 UTC
[PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Jibing-Li opened a new pull request, #31464:
URL: https://github.com/apache/doris/pull/31464
Google caffeine asyncLoad implementation has the logic to avoid repeat call of loading function. Remove the useless code.
backport https://github.com/apache/doris/pull/31380
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1966016592
<details>
<summary>TPC-H: <b>Total hot run time: 49956 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e6f3f236571f2577570325cd694d19398748c6a9, data reload: false
------ Round 1 ----------------------------------
q1 17877 4349 4336 4336
q2 2092 159 139 139
q3 10281 1900 1915 1900
q4 10105 1256 1311 1256
q5 8404 3932 3998 3932
q6 232 121 122 121
q7 2067 1592 1569 1569
q8 9316 2733 2718 2718
q9 10713 10597 10399 10399
q10 8636 3589 3527 3527
q11 417 247 243 243
q12 464 304 296 296
q13 18350 3961 4047 3961
q14 347 320 319 319
q15 503 450 455 450
q16 704 598 598 598
q17 1128 950 974 950
q18 7314 6940 6965 6940
q19 1670 1502 1495 1495
q20 514 292 306 292
q21 4477 4114 4537 4114
q22 499 422 401 401
Total cold run time: 116110 ms
Total hot run time: 49956 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4349 4323 4290 4290
q2 318 229 218 218
q3 4177 4152 4165 4152
q4 2757 2741 2759 2741
q5 7285 7203 7261 7203
q6 234 123 118 118
q7 3287 2829 2841 2829
q8 4400 4506 4516 4506
q9 17148 16927 16927 16927
q10 4226 4282 4284 4282
q11 768 687 696 687
q12 1032 862 873 862
q13 6841 3723 3721 3721
q14 439 431 415 415
q15 496 452 444 444
q16 763 694 691 691
q17 3850 3812 3811 3811
q18 8793 8711 8853 8711
q19 1718 1706 1663 1663
q20 2423 2183 2115 2115
q21 8606 8499 8599 8499
q22 1047 981 966 966
Total cold run time: 84957 ms
Total hot run time: 79851 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1966044950
<details>
<summary>ClickBench: <b>Total hot run time: 30.91 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e6f3f236571f2577570325cd694d19398748c6a9, data reload: false
query1 0.03 0.02 0.02
query2 0.06 0.02 0.02
query3 0.25 0.04 0.05
query4 1.83 0.07 0.07
query5 0.54 0.53 0.52
query6 1.25 0.62 0.62
query7 0.01 0.01 0.01
query8 0.03 0.02 0.02
query9 0.52 0.48 0.49
query10 0.53 0.55 0.53
query11 0.11 0.08 0.08
query12 0.11 0.09 0.09
query13 0.61 0.60 0.62
query14 0.79 0.79 0.80
query15 0.77 0.77 0.77
query16 0.35 0.35 0.37
query17 1.00 1.00 1.00
query18 0.21 0.25 0.25
query19 1.85 1.85 1.88
query20 0.02 0.02 0.01
query21 15.47 0.55 0.56
query22 1.86 2.13 1.46
query23 17.29 1.02 0.95
query24 8.45 1.16 1.40
query25 1.55 0.11 0.11
query26 0.41 0.14 0.15
query27 0.11 0.11 0.10
query28 4.99 0.73 0.74
query29 12.67 2.32 2.26
query30 0.62 0.57 0.53
query31 2.82 0.38 0.41
query32 3.40 0.50 0.49
query33 3.07 3.09 3.06
query34 15.25 4.82 4.80
query35 4.82 4.82 4.84
query36 1.07 1.02 1.01
query37 0.05 0.05 0.05
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.13 0.14
query41 0.07 0.01 0.01
query42 0.02 0.01 0.02
query43 0.02 0.01 0.02
Total cold run time: 105.09 s
Total hot run time: 30.91 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1966050509
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit e6f3f236571f2577570325cd694d19398748c6a9 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.2 seconds inserted 10000000 Rows, about 471K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1966283260
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li merged PR #31464:
URL: https://github.com/apache/doris/pull/31464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1965985919
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "Jibing-Li (via GitHub)" <gi...@apache.org>.
Jibing-Li commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1965987514
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](statistics) Remove useless async loader code. (#31380) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #31464:
URL: https://github.com/apache/doris/pull/31464#issuecomment-1966036137
<details>
<summary>TPC-DS: <b>Total hot run time: 201734 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e6f3f236571f2577570325cd694d19398748c6a9, data reload: false
query1 936 392 377 377
query2 6538 2244 2079 2079
query3 6917 209 201 201
query4 21372 18027 17950 17950
query5 19727 6509 6620 6509
query6 286 216 224 216
query7 4156 296 290 290
query8 251 224 225 224
query9 3130 2696 2609 2609
query10 408 292 295 292
query11 11324 10763 10804 10763
query12 119 73 71 71
query13 5582 627 600 600
query14 17734 13531 13319 13319
query15 363 227 233 227
query16 6468 258 258 258
query17 1728 1449 859 859
query18 2351 406 397 397
query19 200 137 147 137
query20 79 74 71 71
query21 185 99 91 91
query22 5320 5220 5171 5171
query23 32488 31781 32020 31781
query24 7003 6549 6589 6549
query25 514 418 420 418
query26 534 154 156 154
query27 1875 288 291 288
query28 6126 2211 2171 2171
query29 2891 2656 2780 2656
query30 236 161 162 161
query31 923 723 700 700
query32 67 57 59 57
query33 389 251 240 240
query34 841 470 462 462
query35 1104 908 950 908
query36 1714 1451 1414 1414
query37 86 62 57 57
query38 3093 2936 2911 2911
query39 1339 1323 1327 1323
query40 202 90 92 90
query41 37 31 31 31
query42 92 90 82 82
query43 602 758 563 563
query44 1110 704 706 704
query45 238 226 224 224
query46 1230 966 974 966
query47 1757 1907 1838 1838
query48 957 687 635 635
query49 610 375 360 360
query50 856 590 594 590
query51 4764 4599 4722 4599
query52 88 82 69 69
query53 443 309 305 305
query54 2662 2458 2424 2424
query55 89 79 85 79
query56 206 206 199 199
query57 1232 1068 1051 1051
query58 213 208 181 181
query59 3537 3149 3395 3149
query60 202 189 189 189
query61 80 82 81 81
query62 877 490 447 447
query63 472 324 326 324
query64 2264 1452 1394 1394
query65 3628 3575 3575 3575
query66 817 352 351 351
query67 15819 15456 16766 15456
query68 8594 645 652 645
query69 565 340 333 333
query70 2001 1604 1865 1604
query71 397 291 302 291
query72 6441 3389 3407 3389
query73 724 314 319 314
query74 6293 5961 5926 5926
query75 5219 3679 3635 3635
query76 5228 1172 1197 1172
query77 874 247 244 244
query78 12696 11569 11827 11569
query79 7377 629 636 629
query80 1007 397 393 393
query81 495 229 228 228
query82 1503 98 99 98
query83 158 135 129 129
query84 251 68 68 68
query85 859 271 274 271
query86 417 364 384 364
query87 3246 3035 3021 3021
query88 4882 2344 2354 2344
query89 392 299 304 299
query90 1942 209 218 209
query91 147 123 127 123
query92 55 49 48 48
query93 4397 577 601 577
query94 708 205 208 205
query95 1105 1095 1061 1061
query96 646 338 335 335
query97 6409 6405 6287 6287
query98 191 172 170 170
query99 2891 1011 836 836
Total cold run time: 309591 ms
Total hot run time: 201734 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org