You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zhangstar333 (via GitHub)" <gi...@apache.org> on 2023/12/11 08:01:01 UTC
[PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
zhangstar333 opened a new pull request, #28243:
URL: https://github.com/apache/doris/pull/28243
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1849587366
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.64 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17219757943 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1849572304
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit b13a564c7ef1bfe51cd89963f22da8dc5e0faf4b, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4715 4459 4463 4459
q2 372 143 161 143
q3 1465 1213 1222 1213
q4 1109 881 821 821
q5 3155 3145 3148 3145
q6 249 128 127 127
q7 963 490 490 490
q8 2325 2185 2166 2166
q9 6698 6660 6659 6659
q10 3190 3234 3256 3234
q11 330 215 202 202
q12 349 207 204 204
q13 4527 3808 3871 3808
q14 246 214 219 214
q15 564 528 533 528
q16 445 387 383 383
q17 1003 570 537 537
q18 8287 7463 8077 7463
q19 1506 1295 1470 1295
q20 647 303 302 302
q21 3071 2649 2758 2649
q22 349 278 284 278
Total cold run time: 45565 ms
Total hot run time: 40320 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4388 4389 4361 4361
q2 267 163 174 163
q3 3558 3552 3539 3539
q4 2378 2369 2367 2367
q5 5736 5742 5742 5742
q6 242 123 123 123
q7 2383 1865 1845 1845
q8 3538 3538 3532 3532
q9 9084 8961 9035 8961
q10 3864 3966 3988 3966
q11 492 383 396 383
q12 768 592 593 592
q13 4285 3562 3552 3552
q14 289 245 253 245
q15 571 512 521 512
q16 517 468 485 468
q17 1889 1884 1863 1863
q18 8704 8312 8426 8312
q19 1730 1764 1743 1743
q20 2253 1938 1941 1938
q21 6548 6205 6194 6194
q22 506 421 424 421
Total cold run time: 63990 ms
Total hot run time: 60822 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1849516528
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #28243:
URL: https://github.com/apache/doris/pull/28243#discussion_r1422235972
##########
be/src/vec/exec/scan/scanner_context.h:
##########
@@ -161,6 +161,9 @@ class ScannerContext {
int thread_slot_num = 0;
thread_slot_num = (allowed_blocks_num() + _block_per_scanner - 1) / _block_per_scanner;
thread_slot_num = std::min(thread_slot_num, _max_thread_num - _num_running_scanners);
Review Comment:
thread_slot_num = std::max(std::min(thread_slot_num, _max_thread_num - _num_running_scanners), 1);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1849519625
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1851212556
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "zhangstar333 (via GitHub)" <gi...@apache.org>.
zhangstar333 commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1849970440
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #28243:
URL: https://github.com/apache/doris/pull/28243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1850062760
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit f4fcf6b1bc1d967ae927b1cee596d0df0520b239, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4705 4475 4449 4449
q2 359 156 158 156
q3 1447 1229 1240 1229
q4 1097 935 924 924
q5 3116 3141 3151 3141
q6 248 128 127 127
q7 990 488 475 475
q8 2212 2197 2180 2180
q9 6670 6658 6633 6633
q10 3201 3238 3259 3238
q11 331 202 192 192
q12 349 204 210 204
q13 4574 3914 3801 3801
q14 250 209 213 209
q15 574 522 525 522
q16 440 381 383 381
q17 1000 559 542 542
q18 7620 7845 7135 7135
q19 1520 1285 1400 1285
q20 1774 320 298 298
q21 3059 2660 2670 2660
q22 346 275 285 275
Total cold run time: 45882 ms
Total hot run time: 40056 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4437 4395 4441 4395
q2 269 168 171 168
q3 3548 3554 3537 3537
q4 2375 2368 2364 2364
q5 5750 5727 5742 5727
q6 239 118 122 118
q7 2419 1881 1891 1881
q8 3510 3523 3511 3511
q9 9051 8982 9072 8982
q10 3884 4002 3994 3994
q11 517 376 373 373
q12 766 642 588 588
q13 4297 3589 3579 3579
q14 283 250 256 250
q15 565 515 515 515
q16 512 466 524 466
q17 1875 1854 1856 1854
q18 8808 8347 9453 8347
q19 1714 1736 1729 1729
q20 2267 1931 1923 1923
q21 6536 6167 6193 6167
q22 498 437 429 429
Total cold run time: 64120 ms
Total hot run time: 60897 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1849588841
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1850083310
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.92 seconds
stream load tsv: 589 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17219952702 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Bug](scan)fix some case query timeout of not schedule scanner [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28243:
URL: https://github.com/apache/doris/pull/28243#issuecomment-1851212527
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org