You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "HHoflittlefish777 (via GitHub)" <gi...@apache.org> on 2024/01/22 06:19:17 UTC
[PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
HHoflittlefish777 opened a new pull request, #30211:
URL: https://github.com/apache/doris/pull/30211
## Proposed changes
#30136
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #30211:
URL: https://github.com/apache/doris/pull/30211#issuecomment-1903336614
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #30211:
URL: https://github.com/apache/doris/pull/30211
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30211:
URL: https://github.com/apache/doris/pull/30211#issuecomment-1903388279
<details>
<summary>TPC-DS: <b>Total hot run time: 239197 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02e4fd96a2604da9fa136fbfefdb01d35feb67d8, data reload: false
query1 910 390 376 376
query2 6539 2232 2294 2232
query3 6927 199 192 192
query4 19786 18110 18232 18110
query5 19725 6280 6297 6280
query6 285 214 228 214
query7 4152 298 302 298
query8 253 233 254 233
query9 3116 2705 2629 2629
query10 427 286 312 286
query11 12006 11491 11424 11424
query12 140 77 74 74
query13 5881 683 650 650
query14 19718 13479 13502 13479
query15 363 243 230 230
query16 6814 271 263 263
query17 3233 885 880 880
query18 2261 414 403 403
query19 214 146 149 146
query20 76 74 76 74
query21 183 98 99 98
query22 5345 5125 5111 5111
query23 32315 31814 31847 31814
query24 6739 6461 6540 6461
query25 519 424 426 424
query26 522 158 157 157
query27 1772 300 299 299
query28 6114 2257 2227 2227
query29 2886 2818 2686 2686
query30 249 163 160 160
query31 900 731 720 720
query32 67 56 56 56
query33 411 254 244 244
query34 853 458 482 458
query35 1144 932 827 827
query36 1609 1491 1447 1447
query37 90 57 60 57
query38 3132 2923 2934 2923
query39 1360 1329 1311 1311
query40 202 99 97 97
query41 35 32 34 32
query42 84 84 90 84
query43 647 644 562 562
query44 1119 715 718 715
query45 246 232 236 232
query46 1240 986 973 973
query47 1927 1862 1813 1813
query48 971 688 673 673
query49 609 367 352 352
query50 854 638 618 618
query51 5726 5556 5664 5556
query52 89 78 77 77
query53 434 331 308 308
query54 2394 2267 2256 2256
query55 85 71 86 71
query56 228 192 206 192
query57 1219 1160 1127 1127
query58 208 186 194 186
query59 3228 3228 3220 3220
query60 206 202 188 188
query61 84 84 89 84
query62 829 477 425 425
query63 475 332 332 332
query64 1997 1337 1312 1312
query65 3677 3620 3600 3600
query66 825 376 361 361
query67 15949 17455 15556 15556
query68 6382 655 658 655
query69 574 340 338 338
query70 1786 1580 2049 1580
query71 424 294 315 294
query72 4375 3379 3342 3342
query73 714 317 315 315
query74 6408 5855 5847 5847
query75 4487 3625 3428 3428
query76 4129 1198 1221 1198
query77 554 253 248 248
query78 31717 53184 47499 47499
query79 16175 660 644 644
query80 5135 406 391 391
query81 564 229 224 224
query82 1036 100 100 100
query83 409 145 135 135
query84 266 69 67 67
query85 2587 284 274 274
query86 502 373 405 373
query87 3269 3015 2993 2993
query88 7006 2334 2315 2315
query89 462 282 276 276
query90 2505 217 209 209
query91 159 124 121 121
query92 66 50 50 50
query93 6445 631 564 564
query94 1859 203 205 203
query95 1108 1076 1063 1063
query96 657 326 308 308
query97 6464 6161 6374 6161
query98 179 177 173 173
query99 3939 854 857 854
Total cold run time: 346965 ms
Total hot run time: 239197 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30211:
URL: https://github.com/apache/doris/pull/30211#issuecomment-1903393995
<details>
<summary>ClickBench: <b>Total hot run time: 31.14 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02e4fd96a2604da9fa136fbfefdb01d35feb67d8, data reload: false
query1 0.03 0.02 0.02
query2 0.07 0.02 0.02
query3 0.24 0.05 0.05
query4 1.82 0.07 0.06
query5 0.54 0.52 0.52
query6 1.23 0.62 0.63
query7 0.02 0.01 0.01
query8 0.04 0.02 0.02
query9 0.52 0.49 0.48
query10 0.53 0.53 0.54
query11 0.11 0.09 0.09
query12 0.12 0.09 0.09
query13 0.63 0.62 0.62
query14 0.79 0.80 0.79
query15 0.77 0.76 0.75
query16 0.37 0.37 0.37
query17 1.01 1.01 1.02
query18 0.24 0.24 0.25
query19 1.92 1.89 1.85
query20 0.02 0.01 0.01
query21 15.47 0.59 0.57
query22 1.88 1.97 1.39
query23 17.17 0.97 0.98
query24 4.17 3.33 1.44
query25 2.17 0.18 0.15
query26 0.16 0.13 0.14
query27 0.16 0.16 0.14
query28 6.96 0.78 0.70
query29 12.71 2.29 2.24
query30 0.61 0.47 0.52
query31 2.81 0.38 0.38
query32 3.37 0.50 0.48
query33 3.05 3.08 3.11
query34 15.25 4.78 4.80
query35 4.84 4.85 4.83
query36 1.06 1.00 1.01
query37 0.06 0.04 0.05
query38 0.04 0.02 0.02
query39 0.02 0.01 0.02
query40 0.17 0.14 0.14
query41 0.07 0.01 0.01
query42 0.02 0.01 0.01
query43 0.02 0.01 0.01
Total cold run time: 103.26 s
Total hot run time: 31.14 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30211:
URL: https://github.com/apache/doris/pull/30211#issuecomment-1903397318
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 02e4fd96a2604da9fa136fbfefdb01d35feb67d8 with default session variables
Stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc: 58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 21.1 seconds inserted 10000000 Rows, about 473K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-2.0](routine-load) check offset when schedule tasks [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #30211:
URL: https://github.com/apache/doris/pull/30211#issuecomment-1903373507
<details>
<summary>TPC-H: <b>Total hot run time: 49789 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02e4fd96a2604da9fa136fbfefdb01d35feb67d8, data reload: false
------ Round 1 ----------------------------------
q1 17784 4412 4387 4387
q2 2061 149 141 141
q3 10466 1909 1918 1909
q4 10360 1251 1317 1251
q5 8380 3952 3980 3952
q6 228 123 123 123
q7 2036 1601 1599 1599
q8 9517 2708 2695 2695
q9 10634 10352 10355 10352
q10 8647 3535 3557 3535
q11 424 230 244 230
q12 462 293 304 293
q13 18340 3982 4065 3982
q14 341 323 331 323
q15 495 440 448 440
q16 678 591 581 581
q17 1110 968 945 945
q18 7317 6811 6789 6789
q19 1641 1607 1494 1494
q20 538 304 310 304
q21 4439 4136 4071 4071
q22 498 393 398 393
Total cold run time: 116396 ms
Total hot run time: 49789 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4338 4296 4291 4291
q2 317 220 223 220
q3 4114 4140 4152 4140
q4 2756 2728 2715 2715
q5 7203 7150 7162 7150
q6 233 121 117 117
q7 3238 2833 2826 2826
q8 4869 4558 4532 4532
q9 17189 17071 16897 16897
q10 4267 4220 4273 4220
q11 769 678 689 678
q12 1002 866 857 857
q13 6814 3742 3744 3742
q14 450 417 420 417
q15 489 453 460 453
q16 758 690 709 690
q17 3870 3904 3850 3850
q18 8690 8780 8783 8780
q19 1709 1685 1664 1664
q20 2380 2143 2091 2091
q21 8495 8505 8433 8433
q22 1015 943 934 934
Total cold run time: 84965 ms
Total hot run time: 79697 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org