You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "freemandealer (via GitHub)" <gi...@apache.org> on 2024/07/25 08:41:16 UTC
[PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
freemandealer opened a new pull request, #38372:
URL: https://github.com/apache/doris/pull/38372
Current scanner is spread in both Scan & RScan thread pool. Unify them to get better performance (+10% in TPCH1T & SSB1T bench).
Before, two thread pools are nothing working:
After, only local pool is working
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2250012812
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249794519
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249801927
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249797625
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249873285
<details>
<summary>ClickBench: <b>Total hot run time: 30.54 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d40a128e4720f1fc538fcbc4e46db0187ea9a96a, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.22 0.05 0.05
query4 1.68 0.07 0.08
query5 0.50 0.51 0.49
query6 1.14 0.73 0.73
query7 0.02 0.02 0.01
query8 0.05 0.04 0.04
query9 0.55 0.50 0.51
query10 0.53 0.55 0.54
query11 0.15 0.12 0.11
query12 0.14 0.12 0.13
query13 0.60 0.58 0.59
query14 0.75 0.78 0.78
query15 0.84 0.81 0.82
query16 0.37 0.34 0.37
query17 1.02 1.03 1.00
query18 0.22 0.22 0.21
query19 1.90 1.82 1.70
query20 0.01 0.01 0.01
query21 15.40 0.77 0.67
query22 4.39 7.16 1.85
query23 18.28 1.35 1.28
query24 2.19 0.23 0.21
query25 0.15 0.09 0.08
query26 0.30 0.22 0.21
query27 0.45 0.23 0.22
query28 13.27 1.02 1.00
query29 12.63 3.35 3.33
query30 0.25 0.06 0.05
query31 2.87 0.40 0.39
query32 3.26 0.48 0.48
query33 2.98 2.85 2.94
query34 17.03 4.32 4.35
query35 4.41 4.42 4.46
query36 0.66 0.47 0.47
query37 0.19 0.15 0.16
query38 0.15 0.15 0.14
query39 0.05 0.04 0.03
query40 0.16 0.12 0.13
query41 0.09 0.04 0.05
query42 0.06 0.04 0.04
query43 0.05 0.03 0.04
Total cold run time: 110.08 s
Total hot run time: 30.54 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249837107
<details>
<summary>TPC-H: <b>Total hot run time: 39590 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d40a128e4720f1fc538fcbc4e46db0187ea9a96a, data reload: false
------ Round 1 ----------------------------------
q1 17953 4429 4344 4344
q2 2345 201 197 197
q3 10441 1222 1090 1090
q4 10139 737 739 737
q5 7509 2721 2683 2683
q6 220 140 139 139
q7 967 603 599 599
q8 9235 1921 1940 1921
q9 8945 6553 6602 6553
q10 8726 3761 3867 3761
q11 463 259 241 241
q12 409 225 228 225
q13 17795 2993 2989 2989
q14 290 241 249 241
q15 538 492 483 483
q16 492 383 387 383
q17 965 710 659 659
q18 8083 7530 7438 7438
q19 5227 1065 1075 1065
q20 703 330 338 330
q21 5035 3222 3257 3222
q22 350 290 296 290
Total cold run time: 116830 ms
Total hot run time: 39590 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4411 4254 4251 4251
q2 384 265 270 265
q3 2991 2789 2756 2756
q4 1858 1564 1640 1564
q5 5293 5314 5308 5308
q6 222 134 134 134
q7 2131 1732 1767 1732
q8 3197 3364 3328 3328
q9 8486 8423 8423 8423
q10 3891 3757 3674 3674
q11 603 486 508 486
q12 779 607 603 603
q13 17591 2954 2987 2954
q14 298 269 274 269
q15 514 476 469 469
q16 461 439 435 435
q17 1810 1522 1480 1480
q18 7606 7604 7465 7465
q19 1675 1590 1530 1530
q20 1973 1826 1810 1810
q21 5026 4805 4696 4696
q22 580 539 508 508
Total cold run time: 71780 ms
Total hot run time: 54140 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "freemandealer (via GitHub)" <gi...@apache.org>.
freemandealer commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2250017361
run cloud_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249789055
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249794453
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "gavinchou (via GitHub)" <gi...@apache.org>.
gavinchou merged PR #38372:
URL: https://github.com/apache/doris/pull/38372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhancement](cloud) unify scanner thread pool as LOCAL in cloud mode [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #38372:
URL: https://github.com/apache/doris/pull/38372#issuecomment-2249861564
<details>
<summary>TPC-DS: <b>Total hot run time: 172331 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d40a128e4720f1fc538fcbc4e46db0187ea9a96a, data reload: false
query1 911 368 363 363
query2 6455 1835 1742 1742
query3 6664 204 210 204
query4 28635 17506 17319 17319
query5 4179 476 493 476
query6 281 184 161 161
query7 4596 291 301 291
query8 242 192 189 189
query9 8454 2419 2413 2413
query10 419 278 266 266
query11 10719 10100 9933 9933
query12 132 83 84 83
query13 1630 379 365 365
query14 9767 8247 7018 7018
query15 223 168 171 168
query16 7414 479 507 479
query17 1621 564 550 550
query18 1686 289 292 289
query19 196 148 147 147
query20 92 86 84 84
query21 208 101 98 98
query22 4330 4121 3990 3990
query23 34036 33593 33410 33410
query24 12016 2865 2925 2865
query25 662 395 394 394
query26 1614 153 151 151
query27 2622 279 283 279
query28 6788 2031 2018 2018
query29 985 438 436 436
query30 296 151 149 149
query31 978 760 752 752
query32 98 57 57 57
query33 808 346 339 339
query34 881 472 483 472
query35 907 736 716 716
query36 1136 951 940 940
query37 144 81 80 80
query38 2835 2784 2760 2760
query39 858 792 813 792
query40 266 114 120 114
query41 50 49 46 46
query42 117 96 101 96
query43 515 465 466 465
query44 1227 723 727 723
query45 207 182 177 177
query46 1096 723 718 718
query47 1865 1789 1771 1771
query48 378 300 310 300
query49 1183 425 435 425
query50 804 404 418 404
query51 6679 6738 6679 6679
query52 100 94 91 91
query53 257 189 186 186
query54 1249 445 451 445
query55 80 75 76 75
query56 292 269 280 269
query57 1167 1063 1098 1063
query58 261 264 255 255
query59 2898 2615 2672 2615
query60 306 284 282 282
query61 93 112 101 101
query62 842 655 638 638
query63 215 184 179 179
query64 10669 2220 1739 1739
query65 3172 3104 3082 3082
query66 1339 336 339 336
query67 15873 15196 14999 14999
query68 6565 567 564 564
query69 700 410 346 346
query70 1130 1151 1144 1144
query71 534 273 267 267
query72 8860 5610 5601 5601
query73 780 331 329 329
query74 6268 5679 5615 5615
query75 4632 2683 2678 2678
query76 4710 921 890 890
query77 793 322 311 311
query78 9605 9077 8997 8997
query79 8524 518 535 518
query80 1226 480 482 480
query81 589 221 229 221
query82 1486 134 134 134
query83 336 221 181 181
query84 268 78 78 78
query85 1402 312 298 298
query86 441 309 303 303
query87 3334 3109 3075 3075
query88 4821 2387 2427 2387
query89 556 294 281 281
query90 2071 192 187 187
query91 127 100 101 100
query92 63 49 49 49
query93 7458 539 539 539
query94 970 314 290 290
query95 360 257 256 256
query96 617 269 276 269
query97 3226 2972 3074 2972
query98 228 198 192 192
query99 1576 1313 1268 1268
Total cold run time: 303974 ms
Total hot run time: 172331 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org