You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xzj7019 (via GitHub)" <gi...@apache.org> on 2024/06/04 11:59:23 UTC
[PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
xzj7019 opened a new pull request, #35795:
URL: https://github.com/apache/doris/pull/35795
## Proposed changes
Issue Number: close #xxx
Based on real testing perf. data, the doris_scanner_thread_pool_thread_num's setting will have big influence on multi-stream perf. scenarios. Current 2 * num_cores configuration will be too big on the high configuration cluster be, e.g, per-node with 64 cores, under multi-stream testing cases. Perf. data shows that 5 be cluster with 8 streams ad-hoc test cases will have 60% improvement if reduce this configuration from 2 * num_cores to num_cores.
This change will be initial adjustment and should have a more complete adaptive strategy in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147434068
<details>
<summary>TPC-H: <b>Total hot run time: 40820 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7bf8bebd0abe15ba5726cb138f02d494bd725cd, data reload: false
------ Round 1 ----------------------------------
q1 17642 4353 4275 4275
q2 2041 198 201 198
q3 10440 1219 1123 1123
q4 10224 843 789 789
q5 7465 2676 2712 2676
q6 223 134 137 134
q7 955 600 602 600
q8 9227 2108 2108 2108
q9 9327 6650 6675 6650
q10 9401 3961 3873 3873
q11 478 254 240 240
q12 533 241 250 241
q13 18054 3198 3257 3198
q14 254 213 221 213
q15 509 477 485 477
q16 509 387 392 387
q17 955 563 714 563
q18 8316 7731 7737 7731
q19 7915 1467 1433 1433
q20 698 327 328 327
q21 5200 3342 3257 3257
q22 388 328 327 327
Total cold run time: 120754 ms
Total hot run time: 40820 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4632 4441 4461 4441
q2 374 268 283 268
q3 3163 2951 2971 2951
q4 1911 1588 1602 1588
q5 5480 5545 5457 5457
q6 231 123 126 123
q7 2193 1796 1826 1796
q8 3231 3371 3355 3355
q9 8610 8600 8693 8600
q10 4035 3850 3714 3714
q11 595 489 492 489
q12 822 618 618 618
q13 17103 3156 3113 3113
q14 323 283 265 265
q15 523 480 480 480
q16 492 423 444 423
q17 1796 1476 1509 1476
q18 8029 7676 7395 7395
q19 1764 1590 1656 1590
q20 3025 1781 1774 1774
q21 7096 4774 4722 4722
q22 619 569 581 569
Total cold run time: 76047 ms
Total hot run time: 55207 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 closed pull request #35795: [opt](scanner) reduce local scanner thread pool thread num
URL: https://github.com/apache/doris/pull/35795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147366774
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144474754
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144474681
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 closed pull request #35795: [opt](scanner) reduce default scanner thread pool thread num
URL: https://github.com/apache/doris/pull/35795
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147467208
<details>
<summary>ClickBench: <b>Total hot run time: 30.77 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7bf8bebd0abe15ba5726cb138f02d494bd725cd, data reload: false
query1 0.03 0.03 0.03
query2 0.08 0.03 0.03
query3 0.23 0.06 0.05
query4 1.69 0.10 0.10
query5 0.50 0.47 0.49
query6 1.12 0.72 0.73
query7 0.02 0.01 0.02
query8 0.05 0.04 0.04
query9 0.54 0.48 0.47
query10 0.54 0.54 0.54
query11 0.16 0.11 0.12
query12 0.14 0.12 0.11
query13 0.60 0.59 0.60
query14 0.78 0.77 0.79
query15 0.83 0.80 0.81
query16 0.36 0.35 0.36
query17 1.00 1.01 1.02
query18 0.24 0.25 0.24
query19 1.76 1.74 1.71
query20 0.02 0.01 0.01
query21 15.72 0.66 0.66
query22 4.10 7.35 2.08
query23 18.28 1.34 1.22
query24 2.00 0.24 0.22
query25 0.14 0.08 0.09
query26 0.28 0.18 0.18
query27 0.07 0.07 0.08
query28 13.18 1.02 0.99
query29 13.22 3.39 3.29
query30 0.24 0.06 0.05
query31 2.86 0.39 0.38
query32 3.30 0.46 0.46
query33 2.90 2.90 2.90
query34 17.18 4.49 4.45
query35 4.49 4.51 4.59
query36 0.71 0.48 0.48
query37 0.18 0.15 0.15
query38 0.15 0.15 0.15
query39 0.05 0.03 0.04
query40 0.16 0.14 0.14
query41 0.09 0.05 0.04
query42 0.05 0.04 0.04
query43 0.04 0.04 0.04
Total cold run time: 110.08 s
Total hot run time: 30.77 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147456032
<details>
<summary>TPC-DS: <b>Total hot run time: 171693 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7bf8bebd0abe15ba5726cb138f02d494bd725cd, data reload: false
query1 922 378 370 370
query2 6460 2460 2381 2381
query3 6645 210 206 206
query4 18815 17470 17021 17021
query5 4160 452 465 452
query6 253 156 162 156
query7 4594 309 308 308
query8 337 275 276 275
query9 8843 2433 2409 2409
query10 473 298 272 272
query11 10494 10147 9940 9940
query12 131 87 84 84
query13 1651 366 372 366
query14 10067 6977 6321 6321
query15 219 181 186 181
query16 7153 274 263 263
query17 1006 535 508 508
query18 1869 278 274 274
query19 204 160 155 155
query20 99 82 85 82
query21 207 134 134 134
query22 4149 4017 3920 3920
query23 33593 32960 33163 32960
query24 11348 2861 2807 2807
query25 626 364 355 355
query26 1192 160 185 160
query27 3015 312 318 312
query28 7359 2084 2067 2067
query29 986 636 613 613
query30 295 148 150 148
query31 926 769 731 731
query32 97 54 54 54
query33 758 288 287 287
query34 958 494 490 490
query35 748 626 633 626
query36 1097 932 903 903
query37 153 71 70 70
query38 2875 2711 2732 2711
query39 883 809 807 807
query40 208 127 126 126
query41 56 51 51 51
query42 122 95 98 95
query43 593 555 562 555
query44 1253 736 748 736
query45 192 169 160 160
query46 1066 730 762 730
query47 1872 1782 1746 1746
query48 363 297 294 294
query49 995 402 413 402
query50 779 394 395 394
query51 6803 6802 6779 6779
query52 104 94 92 92
query53 356 290 293 290
query54 868 455 473 455
query55 75 72 74 72
query56 284 277 265 265
query57 1146 1070 1037 1037
query58 251 242 264 242
query59 3580 3094 3372 3094
query60 296 276 282 276
query61 90 91 88 88
query62 654 445 464 445
query63 315 293 288 288
query64 9133 2219 1738 1738
query65 3178 3092 3121 3092
query66 960 317 328 317
query67 15327 14687 14819 14687
query68 4492 553 550 550
query69 537 382 337 337
query70 1106 1124 1132 1124
query71 386 291 285 285
query72 7171 5599 5946 5599
query73 733 323 327 323
query74 5899 5525 5490 5490
query75 3356 2687 2674 2674
query76 2412 995 1014 995
query77 696 307 311 307
query78 10291 9681 9673 9673
query79 1950 516 514 514
query80 919 464 465 464
query81 554 225 217 217
query82 1034 109 102 102
query83 239 171 168 168
query84 272 84 86 84
query85 1191 268 279 268
query86 435 376 335 335
query87 3302 3089 3041 3041
query88 3180 2379 2385 2379
query89 469 392 373 373
query90 1785 194 190 190
query91 125 98 96 96
query92 67 54 52 52
query93 1808 524 508 508
query94 1220 194 191 191
query95 408 316 315 315
query96 588 274 273 273
query97 3143 3029 2995 2995
query98 250 220 212 212
query99 1294 855 834 834
Total cold run time: 267891 ms
Total hot run time: 171693 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "xzj7019 (via GitHub)" <gi...@apache.org>.
xzj7019 commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147376387
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce local scanner thread pool thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2147415521
TeamCity be ut coverage result:
Function Coverage: 37.40% (9283/24823)
Line Coverage: 28.73% (76014/264593)
Region Coverage: 28.12% (39388/140086)
Branch Coverage: 24.53% (19923/81224)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a7bf8bebd0abe15ba5726cb138f02d494bd725cd_a7bf8bebd0abe15ba5726cb138f02d494bd725cd/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144469279
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](scanner) reduce default scanner thread pool thread num [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35795:
URL: https://github.com/apache/doris/pull/35795#issuecomment-2144480920
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org