You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/02/07 11:30:18 UTC
[PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
HHoflittlefish777 opened a new pull request, #30967:
URL: https://github.com/apache/doris/pull/30967
## Proposed changes
#30903
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931931299
<details>
<summary>ClickBench: <b>Total hot run time: 30.29 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cc939742641e8856a1185bf653db1f5849205b6f, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.02 0.02
query3 0.25 0.04 0.04
query4 1.83 0.06 0.06
query5 0.53 0.52 0.53
query6 1.26 0.62 0.62
query7 0.01 0.01 0.01
query8 0.04 0.03 0.02
query9 0.53 0.49 0.47
query10 0.52 0.54 0.53
query11 0.11 0.08 0.09
query12 0.11 0.09 0.09
query13 0.62 0.61 0.61
query14 0.78 0.80 0.81
query15 0.77 0.74 0.76
query16 0.38 0.36 0.37
query17 0.98 0.98 1.00
query18 0.21 0.27 0.27
query19 1.89 1.85 1.89
query20 0.01 0.01 0.01
query21 15.46 0.58 0.57
query22 2.20 2.47 1.24
query23 17.24 0.97 0.80
query24 2.85 0.91 1.36
query25 1.43 0.12 0.12
query26 0.21 0.15 0.13
query27 0.10 0.11 0.11
query28 9.68 0.73 0.71
query29 12.68 2.29 2.31
query30 0.54 0.53 0.50
query31 2.81 0.38 0.38
query32 3.42 0.50 0.49
query33 3.06 3.07 3.05
query34 15.28 4.80 4.83
query35 4.88 4.84 4.86
query36 1.06 1.01 1.02
query37 0.06 0.05 0.04
query38 0.04 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.14 0.14
query41 0.06 0.01 0.01
query42 0.02 0.01 0.01
query43 0.02 0.01 0.02
Total cold run time: 104.19 s
Total hot run time: 30.29 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931936504
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit cc939742641e8856a1185bf653db1f5849205b6f with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.3 seconds inserted 10000000 Rows, about 469K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "xiaokang (via GitHub)" <gi...@apache.org>.
xiaokang merged PR #30967:
URL: https://github.com/apache/doris/pull/30967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931848380
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931922590
<details>
<summary>TPC-DS: <b>Total hot run time: 240715 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cc939742641e8856a1185bf653db1f5849205b6f, data reload: false
query1 923 386 387 386
query2 6514 2171 1950 1950
query3 6914 199 201 199
query4 20556 18151 18196 18151
query5 19884 6644 6588 6588
query6 360 224 260 224
query7 4694 310 317 310
query8 273 243 248 243
query9 3155 2719 2702 2702
query10 433 289 324 289
query11 11886 11050 10883 10883
query12 118 78 74 74
query13 5762 650 641 641
query14 20211 13353 13409 13353
query15 403 233 239 233
query16 6411 264 268 264
query17 1664 1460 860 860
query18 2315 400 405 400
query19 199 140 143 140
query20 77 79 74 74
query21 184 96 90 90
query22 5402 5187 5116 5116
query23 32408 31949 31696 31696
query24 6845 6544 6477 6477
query25 516 420 415 415
query26 502 161 158 158
query27 1791 294 294 294
query28 6045 2185 2174 2174
query29 2857 2786 2700 2700
query30 242 166 163 163
query31 898 711 723 711
query32 63 59 60 59
query33 392 244 233 233
query34 837 464 476 464
query35 1132 969 966 966
query36 1562 1448 1395 1395
query37 89 58 55 55
query38 3052 2961 2901 2901
query39 1374 1311 1324 1311
query40 202 91 89 89
query41 35 32 35 32
query42 82 81 85 81
query43 578 584 627 584
query44 1098 712 728 712
query45 236 226 228 226
query46 1232 998 961 961
query47 1804 1637 1714 1637
query48 962 677 658 658
query49 607 362 363 362
query50 860 590 561 561
query51 5553 5515 5472 5472
query52 79 92 70 70
query53 435 316 324 316
query54 2624 2457 2443 2443
query55 80 73 80 73
query56 214 191 205 191
query57 1170 1082 1011 1011
query58 208 209 195 195
query59 3338 3410 3446 3410
query60 213 185 200 185
query61 83 85 85 85
query62 841 562 478 478
query63 469 328 327 327
query64 2468 1390 1482 1390
query65 3647 3549 3560 3549
query66 750 367 350 350
query67 15478 15235 15896 15235
query68 10863 657 679 657
query69 558 328 341 328
query70 2568 1547 1689 1547
query71 425 297 298 297
query72 6479 3377 3422 3377
query73 741 331 326 326
query74 6425 5838 5971 5838
query75 5634 3720 3635 3635
query76 6992 1143 1176 1143
query77 1202 244 238 238
query78 31876 50854 49735 49735
query79 14548 649 623 623
query80 4369 378 384 378
query81 574 232 228 228
query82 996 93 92 92
query83 333 138 126 126
query84 258 67 67 67
query85 1849 286 271 271
query86 476 380 383 380
query87 3259 3007 2975 2975
query88 6766 2270 2274 2270
query89 369 325 271 271
query90 2500 204 189 189
query91 162 121 127 121
query92 62 51 50 50
query93 4489 557 610 557
query94 1663 209 210 209
query95 1137 1067 1070 1067
query96 644 325 323 323
query97 6576 6402 6310 6310
query98 194 185 179 179
query99 3699 929 883 883
Total cold run time: 352975 ms
Total hot run time: 240715 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931848443
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](multi-table-load) reduce routine load job transaction info storage in memory [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30967:
URL: https://github.com/apache/doris/pull/30967#issuecomment-1931897857
<details>
<summary>TPC-H: <b>Total hot run time: 50280 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cc939742641e8856a1185bf653db1f5849205b6f, data reload: false
------ Round 1 ----------------------------------
q1 17677 4381 4340 4340
q2 2048 145 135 135
q3 10427 1923 1914 1914
q4 10382 1209 1276 1209
q5 8430 3934 3981 3934
q6 229 126 127 126
q7 2046 1622 1611 1611
q8 9518 2704 2724 2704
q9 13710 10782 10816 10782
q10 8627 3518 3549 3518
q11 432 244 245 244
q12 468 303 298 298
q13 18345 3943 4045 3943
q14 344 335 343 335
q15 495 463 459 459
q16 706 584 586 584
q17 1119 976 944 944
q18 7132 7040 6961 6961
q19 1658 1536 1487 1487
q20 519 303 299 299
q21 4478 4147 4064 4064
q22 511 403 389 389
Total cold run time: 119301 ms
Total hot run time: 50280 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4343 4349 4303 4303
q2 322 222 228 222
q3 4208 4172 4105 4105
q4 2737 2763 2765 2763
q5 7362 7193 7297 7193
q6 236 120 123 120
q7 3269 2826 2808 2808
q8 4372 4448 4520 4448
q9 17591 17395 17242 17242
q10 4258 4270 4241 4241
q11 750 687 672 672
q12 1016 878 849 849
q13 5402 3753 3755 3753
q14 452 421 442 421
q15 507 455 455 455
q16 779 707 714 707
q17 3877 3823 3899 3823
q18 8795 8671 8718 8671
q19 1722 1675 1677 1675
q20 2433 2192 2150 2150
q21 8585 8595 8627 8595
q22 1065 959 976 959
Total cold run time: 84081 ms
Total hot run time: 80175 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org