You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "sollhui (via GitHub)" <gi...@apache.org> on 2024/07/08 07:14:44 UTC
[PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
sollhui opened a new pull request, #37459:
URL: https://github.com/apache/doris/pull/37459
pick #36632
Most users only care about the size of **max_batch_interval**, but in order to achieve an interval effect, they have to configure **max_batch_rows** and **max_batch_size** according to the characteristics of the data. By adjusting these two default values, users do not need to worry about configuration in most scenarios.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37459:
URL: https://github.com/apache/doris/pull/37459#issuecomment-2213311670
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 0a71674dba45b9629a32a4ae60615d78a420f7cb with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #37459:
URL: https://github.com/apache/doris/pull/37459
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37459:
URL: https://github.com/apache/doris/pull/37459#issuecomment-2213269249
<details>
<summary>TPC-H: <b>Total hot run time: 50089 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a71674dba45b9629a32a4ae60615d78a420f7cb, data reload: false
------ Round 1 ----------------------------------
q1 17635 4341 4320 4320
q2 2081 152 143 143
q3 10467 1896 1951 1896
q4 10364 1259 1346 1259
q5 8899 3861 3929 3861
q6 247 144 131 131
q7 2053 1602 1589 1589
q8 9359 2758 2715 2715
q9 11015 10456 10578 10456
q10 8608 3526 3552 3526
q11 424 249 250 249
q12 465 302 302 302
q13 18349 3987 4033 3987
q14 362 322 320 320
q15 515 454 454 454
q16 681 577 567 567
q17 1137 986 942 942
q18 7296 6940 6940 6940
q19 1745 1678 1559 1559
q20 517 296 315 296
q21 4457 4132 4132 4132
q22 526 447 445 445
Total cold run time: 117202 ms
Total hot run time: 50089 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4331 4292 4292 4292
q2 319 225 226 225
q3 4217 4192 4177 4177
q4 2756 2763 2767 2763
q5 7179 7168 7165 7165
q6 237 124 125 124
q7 3228 2802 2828 2802
q8 4365 4461 4483 4461
q9 17417 17192 16919 16919
q10 4227 4253 4268 4253
q11 748 686 665 665
q12 1036 854 863 854
q13 6930 3730 3743 3730
q14 442 418 425 418
q15 519 453 453 453
q16 725 680 675 675
q17 3805 3828 3849 3828
q18 8766 8783 8842 8783
q19 1733 1710 1633 1633
q20 2361 2151 2119 2119
q21 8477 8440 8506 8440
q22 1044 982 988 982
Total cold run time: 84862 ms
Total hot run time: 79761 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "sollhui (via GitHub)" <gi...@apache.org>.
sollhui commented on PR #37459:
URL: https://github.com/apache/doris/pull/37459#issuecomment-2213214374
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37459:
URL: https://github.com/apache/doris/pull/37459#issuecomment-2213294364
<details>
<summary>TPC-DS: <b>Total hot run time: 205319 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a71674dba45b9629a32a4ae60615d78a420f7cb, data reload: false
query1 924 434 376 376
query2 6529 2736 2555 2555
query3 6918 207 200 200
query4 20980 17912 17995 17912
query5 19734 6466 6529 6466
query6 309 209 226 209
query7 4326 301 300 300
query8 384 398 432 398
query9 3128 2692 2625 2625
query10 420 333 309 309
query11 11337 10780 10737 10737
query12 132 75 77 75
query13 5607 703 690 690
query14 18265 13412 13776 13412
query15 360 249 254 249
query16 6452 283 272 272
query17 1687 1432 883 883
query18 2298 410 420 410
query19 212 150 154 150
query20 79 79 79 79
query21 192 96 93 93
query22 5179 4977 5007 4977
query23 32494 31853 31693 31693
query24 6783 6505 6545 6505
query25 523 437 427 427
query26 529 166 162 162
query27 1863 296 297 296
query28 6193 2376 2321 2321
query29 2823 2871 2667 2667
query30 239 165 170 165
query31 910 748 723 723
query32 75 60 62 60
query33 399 250 249 249
query34 855 487 492 487
query35 1114 900 934 900
query36 1379 1163 1137 1137
query37 88 58 62 58
query38 3082 2961 2993 2961
query39 1388 1320 1321 1320
query40 209 99 96 96
query41 47 47 43 43
query42 86 83 82 82
query43 764 879 648 648
query44 1113 711 727 711
query45 245 238 238 238
query46 1237 958 985 958
query47 1966 1901 1681 1681
query48 1000 737 703 703
query49 627 383 369 369
query50 876 632 583 583
query51 4789 4644 4581 4581
query52 98 90 95 90
query53 450 330 338 330
query54 2629 2444 2482 2444
query55 84 91 82 82
query56 237 230 214 214
query57 1183 1164 1243 1164
query58 225 219 186 186
query59 4015 4027 3665 3665
query60 214 200 208 200
query61 103 97 95 95
query62 807 529 437 437
query63 492 342 347 342
query64 2249 1488 1441 1441
query65 3644 3517 3535 3517
query66 796 384 386 384
query67 16087 16711 17236 16711
query68 8292 651 656 651
query69 557 351 361 351
query70 1457 1409 1265 1265
query71 402 312 325 312
query72 6529 3490 3539 3490
query73 738 317 316 316
query74 6309 5888 5846 5846
query75 4520 3775 3668 3668
query76 4602 1144 1193 1144
query77 559 255 253 253
query78 12878 13092 12904 12904
query79 11027 646 621 621
query80 1636 397 400 397
query81 522 238 238 238
query82 314 96 95 95
query83 173 133 130 130
query84 255 72 70 70
query85 744 339 350 339
query86 366 310 340 310
query87 3258 3001 2981 2981
query88 5199 2301 2321 2301
query89 357 287 309 287
query90 1769 191 207 191
query91 173 146 139 139
query92 60 54 57 54
query93 1829 545 560 545
query94 905 218 214 214
query95 1104 1071 1049 1049
query96 635 324 322 322
query97 6441 6323 6453 6323
query98 185 179 171 171
query99 2886 839 856 839
Total cold run time: 309142 ms
Total hot run time: 205319 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37459:
URL: https://github.com/apache/doris/pull/37459#issuecomment-2213214294
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](routine-load) increase routine load job default max batch size and rows (#36632) [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37459:
URL: https://github.com/apache/doris/pull/37459#issuecomment-2213305017
<details>
<summary>ClickBench: <b>Total hot run time: 30.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a71674dba45b9629a32a4ae60615d78a420f7cb, data reload: false
query1 0.02 0.02 0.02
query2 0.06 0.03 0.02
query3 0.25 0.05 0.04
query4 1.79 0.06 0.06
query5 0.55 0.52 0.52
query6 1.23 0.62 0.62
query7 0.02 0.01 0.01
query8 0.04 0.03 0.02
query9 0.53 0.48 0.46
query10 0.52 0.53 0.54
query11 0.12 0.09 0.09
query12 0.11 0.09 0.09
query13 0.62 0.61 0.61
query14 0.78 0.77 0.78
query15 0.78 0.75 0.74
query16 0.39 0.36 0.36
query17 1.01 0.98 1.01
query18 0.24 0.24 0.23
query19 1.93 1.81 1.76
query20 0.02 0.00 0.01
query21 15.45 0.55 0.56
query22 2.08 2.07 1.75
query23 17.19 1.13 0.85
query24 4.47 0.84 1.06
query25 0.32 0.09 0.05
query26 0.55 0.16 0.15
query27 0.04 0.05 0.04
query28 9.23 0.72 0.77
query29 12.67 2.30 2.28
query30 0.59 0.49 0.48
query31 2.81 0.38 0.36
query32 3.38 0.50 0.50
query33 3.10 3.06 3.12
query34 15.27 4.83 4.80
query35 4.93 4.87 4.87
query36 1.06 1.01 1.00
query37 0.06 0.05 0.04
query38 0.03 0.02 0.02
query39 0.02 0.01 0.02
query40 0.16 0.13 0.14
query41 0.06 0.01 0.02
query42 0.02 0.02 0.01
query43 0.03 0.02 0.02
Total cold run time: 104.53 s
Total hot run time: 30.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org