You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2024/03/28 02:49:24 UTC
[PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner' default value to 16384 [doris]
mrhhsg opened a new pull request, #32939:
URL: https://github.com/apache/doris/pull/32939
## Proposed changes
The previous default value (2M) is too large, which results in reduced parallel performance.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024304473
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner' default value to 16384 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024303371
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024346968
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 9f9bf393a27e6fed6cbfa6a827f93db9547dcedb with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select: 14.3 seconds inserted 10000000 Rows, about 699K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024332840
<details>
<summary>TPC-H: <b>Total hot run time: 38914 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f9bf393a27e6fed6cbfa6a827f93db9547dcedb, data reload: false
------ Round 1 ----------------------------------
q1 17642 4338 4293 4293
q2 2082 203 196 196
q3 10439 1130 1189 1130
q4 10477 816 770 770
q5 7577 2945 2833 2833
q6 214 135 130 130
q7 1030 623 599 599
q8 9255 2066 2057 2057
q9 7790 6908 6825 6825
q10 8570 3534 3518 3518
q11 466 239 235 235
q12 446 216 215 215
q13 18911 2938 2971 2938
q14 286 230 243 230
q15 518 480 482 480
q16 527 413 397 397
q17 970 555 629 555
q18 7422 6888 6721 6721
q19 4679 1451 1468 1451
q20 651 308 304 304
q21 3467 2723 2947 2723
q22 377 314 328 314
Total cold run time: 113796 ms
Total hot run time: 38914 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4347 4128 4182 4128
q2 379 263 270 263
q3 2973 2697 2756 2697
q4 1875 1481 1511 1481
q5 5392 5352 5422 5352
q6 208 121 124 121
q7 2298 1957 1968 1957
q8 3232 3324 3323 3323
q9 8875 8861 8882 8861
q10 3870 3692 3604 3604
q11 585 498 501 498
q12 756 613 593 593
q13 17719 3005 3038 3005
q14 301 271 273 271
q15 512 484 467 467
q16 500 456 455 455
q17 1773 1428 1453 1428
q18 7644 7488 7346 7346
q19 1644 1516 1534 1516
q20 1951 1787 1770 1770
q21 5136 4829 4806 4806
q22 549 469 460 460
Total cold run time: 72519 ms
Total hot run time: 54402 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024341048
<details>
<summary>TPC-DS: <b>Total hot run time: 182629 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f9bf393a27e6fed6cbfa6a827f93db9547dcedb, data reload: false
query1 913 396 374 374
query2 6482 2029 1881 1881
query3 6672 210 210 210
query4 23933 21384 21317 21317
query5 4303 430 421 421
query6 267 176 193 176
query7 4613 300 291 291
query8 236 184 177 177
query9 8402 2442 2429 2429
query10 532 248 258 248
query11 14792 14256 14353 14256
query12 144 94 93 93
query13 1640 375 365 365
query14 10003 7256 7618 7256
query15 246 209 194 194
query16 8008 271 261 261
query17 1927 562 540 540
query18 2027 283 278 278
query19 203 152 153 152
query20 95 88 86 86
query21 205 132 128 128
query22 5038 4828 4837 4828
query23 33403 32873 32915 32873
query24 11737 2838 2854 2838
query25 625 366 389 366
query26 1828 159 158 158
query27 3132 367 360 360
query28 7465 1907 1896 1896
query29 1084 658 634 634
query30 310 160 159 159
query31 996 729 741 729
query32 96 61 55 55
query33 774 271 260 260
query34 1144 491 479 479
query35 864 630 639 630
query36 1050 925 915 915
query37 292 76 77 76
query38 3604 3447 3425 3425
query39 1519 1466 1444 1444
query40 292 137 134 134
query41 52 48 46 46
query42 107 97 96 96
query43 495 457 451 451
query44 1229 727 733 727
query45 275 281 265 265
query46 1105 716 715 715
query47 1925 1892 1847 1847
query48 389 303 310 303
query49 1192 384 382 382
query50 780 397 391 391
query51 6764 6534 6497 6497
query52 111 94 95 94
query53 351 282 289 282
query54 332 253 245 245
query55 85 79 82 79
query56 255 235 233 233
query57 1230 1172 1142 1142
query58 248 219 219 219
query59 2826 2605 2598 2598
query60 293 257 282 257
query61 114 115 111 111
query62 655 451 436 436
query63 313 283 287 283
query64 6880 4196 4167 4167
query65 3175 3085 3043 3043
query66 1445 377 373 373
query67 15635 14980 15162 14980
query68 8211 548 540 540
query69 563 327 328 327
query70 1221 1177 1195 1177
query71 518 293 272 272
query72 6777 2865 2612 2612
query73 743 330 325 325
query74 6862 6436 6445 6436
query75 3883 2274 2308 2274
query76 4977 912 919 912
query77 634 282 275 275
query78 11082 10231 10154 10154
query79 12711 528 529 528
query80 1849 456 437 437
query81 511 238 228 228
query82 745 90 93 90
query83 211 170 171 170
query84 262 84 86 84
query85 1410 274 263 263
query86 399 290 272 272
query87 3746 3590 3563 3563
query88 5200 2469 2456 2456
query89 526 379 381 379
query90 1982 184 182 182
query91 131 112 104 104
query92 62 50 48 48
query93 7373 528 497 497
query94 1248 188 184 184
query95 441 347 335 335
query96 609 264 277 264
query97 2664 2515 2482 2482
query98 232 219 217 217
query99 1293 888 908 888
Total cold run time: 308233 ms
Total hot run time: 182629 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32939:
URL: https://github.com/apache/doris/pull/32939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024304455
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [chore](session_variable) change parallel_scan_min_rows_per_scanner's default value to 16384 [doris]
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #32939:
URL: https://github.com/apache/doris/pull/32939#issuecomment-2024304048
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org