You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "GoGoWen (via GitHub)" <gi...@apache.org> on 2024/06/13 08:40:00 UTC
[PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
GoGoWen opened a new pull request, #36253:
URL: https://github.com/apache/doris/pull/36253
## Proposed changes
make consumer number of routine load task configurable
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165033581
I think config max_consumer_num_per_group is same with consumer_num_per_task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165069754
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165245154
<details>
<summary>ClickBench: <b>Total hot run time: 31.51 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3a96f8fc4fd7f82e26de59c5b9127d779c02b2c, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.23 0.05 0.05
query4 1.67 0.08 0.08
query5 0.53 0.49 0.49
query6 1.14 0.72 0.73
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.55 0.49 0.49
query10 0.54 0.57 0.53
query11 0.16 0.11 0.11
query12 0.14 0.12 0.12
query13 0.59 0.58 0.60
query14 0.77 0.79 0.80
query15 0.84 0.81 0.81
query16 0.35 0.37 0.37
query17 1.01 0.98 1.01
query18 0.23 0.27 0.22
query19 1.84 1.69 1.73
query20 0.02 0.01 0.01
query21 15.40 0.64 0.64
query22 3.81 6.02 2.94
query23 18.31 1.47 1.21
query24 2.16 0.23 0.20
query25 0.17 0.09 0.08
query26 0.27 0.18 0.17
query27 0.08 0.07 0.07
query28 13.18 1.02 1.00
query29 12.59 3.24 3.26
query30 0.26 0.08 0.05
query31 2.85 0.41 0.40
query32 3.22 0.48 0.47
query33 2.90 2.94 2.90
query34 17.19 4.46 4.52
query35 4.52 4.51 4.45
query36 0.63 0.46 0.48
query37 0.17 0.16 0.15
query38 0.16 0.14 0.14
query39 0.04 0.04 0.04
query40 0.16 0.15 0.15
query41 0.09 0.05 0.04
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 109.07 s
Total hot run time: 31.51 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165070687
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on code in PR #36253:
URL: https://github.com/apache/doris/pull/36253#discussion_r1637916589
##########
fe/fe-common/src/main/java/org/apache/doris/common/FeMetaVersion.java:
##########
@@ -86,9 +86,11 @@ public final class FeMetaVersion {
public static final int VERSION_132 = 132;
// for expr serde
public static final int VERSION_133 = 133;
+ // add config for consumer number of routine load task
+ public static final int VERSION_134 = 134;
// note: when increment meta version, should assign the latest version to VERSION_CURRENT
- public static final int VERSION_CURRENT = VERSION_133;
Review Comment:
Why do you change it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165233274
<details>
<summary>TPC-DS: <b>Total hot run time: 170217 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3a96f8fc4fd7f82e26de59c5b9127d779c02b2c, data reload: false
query1 806 373 375 373
query2 6327 2342 2205 2205
query3 6407 215 206 206
query4 19048 17297 17221 17221
query5 3738 480 456 456
query6 251 154 154 154
query7 4465 298 295 295
query8 336 287 283 283
query9 8444 2483 2469 2469
query10 572 340 309 309
query11 10707 10045 10075 10045
query12 139 84 87 84
query13 1568 372 369 369
query14 10040 6670 8098 6670
query15 233 184 187 184
query16 7820 276 270 270
query17 1626 555 527 527
query18 1957 265 270 265
query19 193 153 152 152
query20 94 87 80 80
query21 232 125 126 125
query22 4473 4048 3927 3927
query23 33950 33694 33670 33670
query24 11225 2759 2828 2759
query25 582 350 364 350
query26 708 154 152 152
query27 2238 328 325 325
query28 5903 2123 2106 2106
query29 860 614 606 606
query30 231 160 156 156
query31 960 726 734 726
query32 97 54 57 54
query33 778 311 291 291
query34 878 470 478 470
query35 735 629 637 629
query36 1102 944 942 942
query37 134 71 67 67
query38 2837 2755 2699 2699
query39 861 809 796 796
query40 205 128 126 126
query41 55 50 50 50
query42 121 95 99 95
query43 576 571 549 549
query44 1186 737 729 729
query45 195 164 162 162
query46 1072 763 737 737
query47 1836 1768 1772 1768
query48 375 303 302 302
query49 864 424 426 424
query50 758 404 408 404
query51 6773 6577 6608 6577
query52 106 88 95 88
query53 357 281 292 281
query54 873 455 445 445
query55 74 76 78 76
query56 273 256 271 256
query57 1163 1030 1063 1030
query58 245 235 244 235
query59 3460 3172 3116 3116
query60 293 291 265 265
query61 95 87 87 87
query62 633 463 437 437
query63 314 291 291 291
query64 8542 2198 1734 1734
query65 3333 3107 3153 3107
query66 778 337 337 337
query67 15518 14924 14789 14789
query68 6221 564 562 562
query69 670 504 436 436
query70 1282 1144 1102 1102
query71 440 275 275 275
query72 7782 5564 2807 2807
query73 766 330 330 330
query74 5868 5457 5586 5457
query75 3828 2681 2653 2653
query76 3264 961 929 929
query77 637 295 305 295
query78 10375 9834 9722 9722
query79 3279 519 523 519
query80 1188 457 452 452
query81 526 214 224 214
query82 738 105 101 101
query83 197 172 168 168
query84 272 89 86 86
query85 1258 296 334 296
query86 406 294 336 294
query87 3249 3063 3038 3038
query88 3801 2454 2476 2454
query89 486 387 387 387
query90 1869 194 198 194
query91 128 100 104 100
query92 60 49 52 49
query93 4689 524 508 508
query94 1155 191 195 191
query95 411 330 312 312
query96 627 270 271 270
query97 3209 3031 3024 3024
query98 211 201 193 193
query99 1184 850 835 835
Total cold run time: 273704 ms
Total hot run time: 170217 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #36253: [Enhancement](Load) make consumer number of routine load task configurable
URL: https://github.com/apache/doris/pull/36253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165060913
> > > max_consumer_num_per_group
> >
> >
> > max_consumer_num_per_group for BE scope, consumer_num_per_task for job scope.
>
> It means that you want to control it in create routine load job stmt?
correct!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165206626
<details>
<summary>TPC-H: <b>Total hot run time: 41426 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3a96f8fc4fd7f82e26de59c5b9127d779c02b2c, data reload: false
------ Round 1 ----------------------------------
q1 17615 4337 4266 4266
q2 2018 196 192 192
q3 10451 1134 1119 1119
q4 10192 824 884 824
q5 7448 2712 2780 2712
q6 256 145 134 134
q7 972 609 608 608
q8 9237 2057 2121 2057
q9 8864 6482 6520 6482
q10 8905 3719 3724 3719
q11 442 249 244 244
q12 437 221 222 221
q13 17750 2956 2973 2956
q14 261 222 223 222
q15 522 476 478 476
q16 516 380 381 380
q17 976 664 699 664
q18 8342 7975 7887 7887
q19 7153 1554 1575 1554
q20 692 333 329 329
q21 5125 4041 4104 4041
q22 431 339 340 339
Total cold run time: 118605 ms
Total hot run time: 41426 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4611 4457 4300 4300
q2 378 266 273 266
q3 3148 2910 2894 2894
q4 1966 1648 1666 1648
q5 5491 5532 5493 5493
q6 221 128 128 128
q7 2226 1931 1758 1758
q8 3266 3424 3411 3411
q9 8761 8773 8740 8740
q10 4122 3946 3917 3917
q11 593 497 492 492
q12 863 576 615 576
q13 17009 3060 3126 3060
q14 318 252 295 252
q15 528 472 482 472
q16 486 444 426 426
q17 1835 1516 1542 1516
q18 7949 7733 7779 7733
q19 1773 1681 1501 1501
q20 3046 1883 1885 1883
q21 7367 4820 4863 4820
q22 618 545 560 545
Total cold run time: 76575 ms
Total hot run time: 55831 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "HHoflittlefish777 (via GitHub)" <gi...@apache.org>.
HHoflittlefish777 commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165059302
> > max_consumer_num_per_group
>
> max_consumer_num_per_group for BE scope, consumer_num_per_task for job scope.
It means that you want to control it in create routine load job stmt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165026107
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165011415
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "GoGoWen (via GitHub)" <gi...@apache.org>.
GoGoWen commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2165044931
> max_consumer_num_per_group
max_consumer_num_per_group for BE scope, consumer_num_per_task for job scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Enhancement](Load) make consumer number of routine load task configurable [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36253:
URL: https://github.com/apache/doris/pull/36253#issuecomment-2533298924
We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org