You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2024/07/15 13:28:04 UTC
[PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
freemandealer opened a new pull request, #37848:
URL: https://github.com/apache/doris/pull/37848
When to enable commit on be?
Bypassing FE during commit is good for load performance, especially when doing freqent streamloading.
When to disable commit on be?
The current implementation of bypass will mess up the statistics, thus the query plan optimizer. So disable it when you encoutering any troubles. Such problem will be fixed in future work.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228592773
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228537014
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2229898795
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228717167
<details>
<summary>TPC-H: <b>Total hot run time: 40085 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 63a87ee9a674636b170bc044ed89ec6de94d67d6, data reload: false
------ Round 1 ----------------------------------
q1 17821 4673 4420 4420
q2 2653 204 201 201
q3 11041 1216 1116 1116
q4 10428 801 786 786
q5 7637 2748 2721 2721
q6 225 138 139 138
q7 985 607 612 607
q8 9593 2146 2124 2124
q9 8634 6565 6585 6565
q10 8695 3796 3784 3784
q11 465 233 247 233
q12 402 232 221 221
q13 17794 2942 3004 2942
q14 284 233 242 233
q15 532 481 500 481
q16 481 384 374 374
q17 992 653 735 653
q18 8153 7403 7384 7384
q19 3703 1367 1418 1367
q20 682 329 325 325
q21 4919 3177 3126 3126
q22 356 294 284 284
Total cold run time: 116475 ms
Total hot run time: 40085 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4350 4296 4265 4265
q2 375 262 264 262
q3 2967 2841 2729 2729
q4 1877 1567 1621 1567
q5 5318 5278 5330 5278
q6 221 132 128 128
q7 2120 1700 1764 1700
q8 3231 3347 3327 3327
q9 8486 8427 8383 8383
q10 3936 3694 3658 3658
q11 594 490 486 486
q12 762 618 615 615
q13 16632 2982 2985 2982
q14 296 267 273 267
q15 522 476 472 472
q16 475 418 407 407
q17 1785 1490 1498 1490
q18 7718 7761 7330 7330
q19 1703 1508 1599 1508
q20 1980 1755 1747 1747
q21 4768 4610 4764 4610
q22 572 514 509 509
Total cold run time: 70688 ms
Total hot run time: 53720 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228745443
<details>
<summary>TPC-DS: <b>Total hot run time: 172832 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 63a87ee9a674636b170bc044ed89ec6de94d67d6, data reload: false
query1 927 378 366 366
query2 6464 1888 1836 1836
query3 6668 215 214 214
query4 28383 17723 17236 17236
query5 4245 473 481 473
query6 276 167 170 167
query7 4596 293 282 282
query8 256 198 198 198
query9 8520 2446 2421 2421
query10 450 284 275 275
query11 11211 10059 10140 10059
query12 136 90 85 85
query13 1642 371 368 368
query14 9968 7632 7631 7631
query15 249 168 177 168
query16 8097 336 325 325
query17 1777 557 545 545
query18 2078 286 280 280
query19 199 153 155 153
query20 90 84 82 82
query21 219 132 136 132
query22 4229 4118 3936 3936
query23 33758 32977 33128 32977
query24 12037 2830 2839 2830
query25 675 391 388 388
query26 1777 149 148 148
query27 2964 281 275 275
query28 7660 2037 2020 2020
query29 1072 644 625 625
query30 309 152 149 149
query31 945 745 750 745
query32 100 54 55 54
query33 797 320 304 304
query34 968 484 491 484
query35 684 587 584 584
query36 1111 927 900 900
query37 295 89 81 81
query38 2918 2742 2788 2742
query39 860 816 820 816
query40 279 125 124 124
query41 49 47 48 47
query42 114 101 99 99
query43 504 456 457 456
query44 1210 747 736 736
query45 199 168 166 166
query46 1095 732 710 710
query47 1860 1796 1801 1796
query48 374 311 295 295
query49 1205 422 414 414
query50 781 394 386 386
query51 6872 6898 6825 6825
query52 107 92 98 92
query53 356 294 288 288
query54 989 455 457 455
query55 78 75 76 75
query56 316 290 303 290
query57 1166 1065 1056 1056
query58 269 258 267 258
query59 3000 2615 2522 2522
query60 311 275 267 267
query61 97 96 99 96
query62 827 655 668 655
query63 339 284 290 284
query64 10401 2269 1687 1687
query65 3213 3116 3141 3116
query66 1368 337 335 335
query67 15764 15005 14926 14926
query68 9186 589 567 567
query69 740 465 359 359
query70 1426 1133 1141 1133
query71 563 280 284 280
query72 9136 5913 5658 5658
query73 2232 325 318 318
query74 6030 5653 5728 5653
query75 5507 2726 2729 2726
query76 5553 913 934 913
query77 771 297 293 293
query78 9850 8985 10154 8985
query79 9244 527 533 527
query80 897 482 470 470
query81 577 221 218 218
query82 415 142 135 135
query83 344 166 169 166
query84 280 84 92 84
query85 1036 307 350 307
query86 362 305 318 305
query87 3293 3093 3195 3093
query88 4372 2360 2377 2360
query89 524 377 376 376
query90 2020 201 197 197
query91 129 98 98 98
query92 66 51 54 51
query93 5852 529 515 515
query94 1334 209 214 209
query95 416 322 310 310
query96 603 269 265 265
query97 3182 3013 3010 3010
query98 229 198 205 198
query99 1503 1243 1282 1243
Total cold run time: 310425 ms
Total hot run time: 172832 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228519900
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring closed pull request #37848: [fix](cloud) provide a conf to enable/disable streamload commit on be
URL: https://github.com/apache/doris/pull/37848
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228507112
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228758963
<details>
<summary>ClickBench: <b>Total hot run time: 31.16 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 63a87ee9a674636b170bc044ed89ec6de94d67d6, data reload: false
query1 0.04 0.03 0.04
query2 0.08 0.04 0.05
query3 0.23 0.05 0.05
query4 1.66 0.08 0.08
query5 0.51 0.49 0.52
query6 1.14 0.73 0.73
query7 0.02 0.02 0.02
query8 0.05 0.05 0.04
query9 0.55 0.49 0.49
query10 0.56 0.55 0.55
query11 0.15 0.11 0.11
query12 0.15 0.12 0.13
query13 0.59 0.59 0.59
query14 0.76 0.77 0.77
query15 0.85 0.82 0.82
query16 0.34 0.36 0.37
query17 1.00 1.03 1.02
query18 0.23 0.22 0.22
query19 1.83 1.84 1.75
query20 0.01 0.01 0.01
query21 15.40 0.73 0.64
query22 4.36 6.53 2.34
query23 18.29 1.36 1.23
query24 2.11 0.22 0.23
query25 0.15 0.09 0.10
query26 0.29 0.21 0.22
query27 0.46 0.23 0.24
query28 13.27 1.02 1.00
query29 12.71 3.34 3.34
query30 0.25 0.06 0.05
query31 2.88 0.39 0.38
query32 3.27 0.48 0.47
query33 2.85 2.97 2.88
query34 17.07 4.34 4.33
query35 4.37 4.40 4.42
query36 0.64 0.48 0.48
query37 0.19 0.16 0.17
query38 0.15 0.15 0.14
query39 0.05 0.04 0.04
query40 0.15 0.13 0.12
query41 0.09 0.05 0.04
query42 0.06 0.05 0.05
query43 0.04 0.03 0.04
Total cold run time: 109.85 s
Total hot run time: 31.16 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](cloud) provide a conf to enable/disable streamload commit on be [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37848:
URL: https://github.com/apache/doris/pull/37848#issuecomment-2228592842
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org