You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2025/11/17 03:33:04 UTC
[PR] [Improvement](external) Increase scanner concurrency [doris]
Gabriel39 opened a new pull request, #58073:
URL: https://github.com/apache/doris/pull/58073
### What problem does this PR solve?
The concurrency of scanners are controlled by some variables which is not suitable for different cases. For external tables, we need more scanners than the internal tables. In this PR, the concurrency variable is splitted from the origin one.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562531645
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -31,6 +31,39 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
+
+int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of file scanners for each FileScanLocalState is determined by:
+ * 1. User specified max_file_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(512, CpuInfo::num_cores() * 10)
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_file_scanners_concurrency() ? state->max_file_scanners_concurrency()
+ : std::max(512, CpuInfo::num_cores() * 10) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int FileScanLocalState::min_scanners_concurrency(RuntimeState* state) const {
+ /*
+ * The min concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified min_scan_concurrency_of_scan_scheduler which is set through session variable.
+ * 2. Default: 2 * state->get_query_ctx()->get_remote_scan_scheduler()->get_max_threads()
+ */
+ return state->min_scan_concurrency_of_scan_scheduler()
+ ? state->min_scan_concurrency_of_scan_scheduler()
+ : 2 * state->get_query_ctx()->get_remote_scan_scheduler()->get_max_threads();
Review Comment:
这里换成跟 cpu core的关系，不要依赖max threads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3581702724
# FE Regression Coverage Report
Increment line coverage ` 80.00% (8/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584445244
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3540124012
<details>
<summary>ClickBench: <b>Total hot run time: 28.52 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e0bd88b8407116b6be9c54135c09920f79b3756, data reload: false
query1 0.05 0.04 0.04
query2 0.14 0.07 0.07
query3 0.32 0.08 0.08
query4 1.61 0.10 0.09
query5 0.27 0.27 0.25
query6 1.19 0.64 0.64
query7 0.03 0.03 0.02
query8 0.08 0.06 0.06
query9 0.60 0.53 0.52
query10 0.58 0.58 0.56
query11 0.25 0.14 0.14
query12 0.26 0.15 0.15
query13 0.63 0.64 0.61
query14 1.01 1.01 1.00
query15 0.86 0.82 0.82
query16 0.39 0.41 0.40
query17 1.08 1.06 0.98
query18 0.24 0.23 0.22
query19 1.92 1.91 1.84
query20 0.02 0.02 0.01
query21 15.42 0.29 0.24
query22 5.00 0.12 0.10
query23 15.39 0.36 0.23
query24 2.28 0.45 0.32
query25 0.10 0.10 0.10
query26 0.20 0.18 0.18
query27 0.10 0.09 0.09
query28 3.70 1.26 1.10
query29 12.55 4.03 3.34
query30 0.35 0.13 0.12
query31 2.82 0.64 0.44
query32 3.28 0.60 0.51
query33 3.04 3.06 3.08
query34 16.09 5.14 4.59
query35 4.59 4.61 4.61
query36 0.63 0.51 0.49
query37 0.24 0.09 0.09
query38 0.20 0.05 0.06
query39 0.06 0.04 0.05
query40 0.22 0.17 0.16
query41 0.12 0.08 0.06
query42 0.07 0.05 0.05
query43 0.07 0.05 0.06
Total cold run time: 98.05 s
Total hot run time: 28.52 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3570334601
<details>
<summary>TPC-H: <b>Total hot run time: 34278 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6d4a5a626312a4f4c9a77a0ad5c412dca03620b0, data reload: false
------ Round 1 ----------------------------------
q1 17594 5044 4891 4891
q2 2029 314 199 199
q3 10207 1295 713 713
q4 10239 960 364 364
q5 7554 2531 2263 2263
q6 190 173 139 139
q7 905 773 629 629
q8 9290 1336 1020 1020
q9 7329 5357 5377 5357
q10 6861 2231 1825 1825
q11 500 306 294 294
q12 359 370 228 228
q13 17796 3739 3050 3050
q14 243 241 218 218
q15 572 511 527 511
q16 1030 1011 934 934
q17 593 747 498 498
q18 7378 7181 7144 7144
q19 1102 953 563 563
q20 345 351 228 228
q21 3582 2605 2292 2292
q22 1012 968 918 918
Total cold run time: 106710 ms
Total hot run time: 34278 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4943 4892 4904 4892
q2 332 393 320 320
q3 2133 2695 2298 2298
q4 1305 1758 1293 1293
q5 4187 4314 4422 4314
q6 227 173 135 135
q7 2048 1940 1836 1836
q8 2664 2444 2463 2444
q9 7625 7648 7590 7590
q10 3059 3250 2830 2830
q11 586 515 504 504
q12 693 747 652 652
q13 3720 3849 3352 3352
q14 288 327 289 289
q15 528 510 512 510
q16 1062 1086 1036 1036
q17 1144 1506 1416 1416
q18 7819 7864 7513 7513
q19 766 749 761 749
q20 2157 2039 1911 1911
q21 4769 4249 4193 4193
q22 1051 988 967 967
Total cold run time: 53106 ms
Total hot run time: 51044 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3570695566
# BE UT Coverage Report
Increment line coverage `18.60% (8/43)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6d4a5a626312a4f4c9a77a0ad5c412dca03620b0_6d4a5a626312a4f4c9a77a0ad5c412dca03620b0/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6d4a5a626312a4f4c9a77a0ad5c412dca03620b0_6d4a5a626312a4f4c9a77a0ad5c412dca03620b0/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.63% (18280/34735) |
| Line Coverage | 38.05% (166262/436947) |
| Region Coverage | 33.00% (129229/391573) |
| Branch Coverage | 33.80% (55461/164082) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3542713118
# BE UT Coverage Report
Increment line coverage `20.51% (8/39)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/22249e2f89b2266965806f7444efdeb3e16ab171_22249e2f89b2266965806f7444efdeb3e16ab171/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/22249e2f89b2266965806f7444efdeb3e16ab171_22249e2f89b2266965806f7444efdeb3e16ab171/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.74% (18274/34646) |
| Line Coverage | 38.12% (166087/435738) |
| Region Coverage | 33.08% (129138/390377) |
| Branch Coverage | 33.83% (55395/163759) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3569682292
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2555033634
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -31,6 +31,15 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
+
+int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) {
+ return (state->max_file_scanners_concurrency()
Review Comment:
Add some comment for this logic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562533157
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -31,6 +31,39 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
+
+int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of file scanners for each FileScanLocalState is determined by:
+ * 1. User specified max_file_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(512, CpuInfo::num_cores() * 10)
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_file_scanners_concurrency() ? state->max_file_scanners_concurrency()
Review Comment:
max_file_scanners_concurrency > 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3578475777
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562529484
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -31,6 +31,39 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
+
+int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of file scanners for each FileScanLocalState is determined by:
+ * 1. User specified max_file_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(512, CpuInfo::num_cores() * 10)
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_file_scanners_concurrency() ? state->max_file_scanners_concurrency()
+ : std::max(512, CpuInfo::num_cores() * 10) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int FileScanLocalState::min_scanners_concurrency(RuntimeState* state) const {
+ /*
+ * The min concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified min_scan_concurrency_of_scan_scheduler which is set through session variable.
+ * 2. Default: 2 * state->get_query_ctx()->get_remote_scan_scheduler()->get_max_threads()
+ */
+ return state->min_scan_concurrency_of_scan_scheduler()
Review Comment:
min_scan_concurrency_of_scan_scheduler > 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3578602796
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/98f7f81cbd7268940b4d1428e678607919e50b8c_98f7f81cbd7268940b4d1428e678607919e50b8c_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/98f7f81cbd7268940b4d1428e678607919e50b8c_98f7f81cbd7268940b4d1428e678607919e50b8c_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.86% (1707/2111) |
| Line Coverage | 66.85% (29889/44708) |
| Region Coverage | 67.38% (14900/22114) |
| Branch Coverage | 57.56% (7925/13768) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3575078848
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3580676140
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0_dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0_dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.86% (1707/2111) |
| Line Coverage | 66.84% (29881/44708) |
| Region Coverage | 67.38% (14900/22114) |
| Branch Coverage | 57.58% (7927/13768) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3539931389
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2e0bd88b8407116b6be9c54135c09920f79b3756_2e0bd88b8407116b6be9c54135c09920f79b3756_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2e0bd88b8407116b6be9c54135c09920f79b3756_2e0bd88b8407116b6be9c54135c09920f79b3756_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.56% (1683/2089) |
| Line Coverage | 66.71% (29696/44518) |
| Region Coverage | 67.27% (14819/22028) |
| Branch Coverage | 57.54% (7886/13706) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3542288369
<details>
<summary>TPC-H: <b>Total hot run time: 36006 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22249e2f89b2266965806f7444efdeb3e16ab171, data reload: false
------ Round 1 ----------------------------------
q1 17401 5569 5188 5188
q2 2010 307 206 206
q3 10161 1242 713 713
q4 10225 897 348 348
q5 7498 2443 2354 2354
q6 187 176 137 137
q7 939 761 625 625
q8 9355 1426 1319 1319
q9 7375 5294 5418 5294
q10 6912 2263 1789 1789
q11 501 292 272 272
q12 331 367 220 220
q13 17784 3627 3016 3016
q14 241 236 208 208
q15 569 529 499 499
q16 987 1026 928 928
q17 593 873 342 342
q18 8234 7529 7665 7529
q19 1123 977 597 597
q20 380 369 256 256
q21 4242 3384 3099 3099
q22 1200 1081 1067 1067
Total cold run time: 108248 ms
Total hot run time: 36006 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5489 5571 5486 5486
q2 245 326 230 230
q3 2489 2810 2467 2467
q4 1478 1879 1461 1461
q5 4571 4948 5022 4948
q6 177 160 126 126
q7 2014 1963 2009 1963
q8 2543 2828 2623 2623
q9 7402 7527 7230 7230
q10 2874 3167 2623 2623
q11 573 504 497 497
q12 625 709 574 574
q13 3267 3651 3001 3001
q14 272 273 253 253
q15 526 503 477 477
q16 1021 1043 979 979
q17 1180 1694 1219 1219
q18 7288 7261 7141 7141
q19 760 988 1060 988
q20 1909 1940 1823 1823
q21 4758 4399 4297 4297
q22 1098 1039 995 995
Total cold run time: 52559 ms
Total hot run time: 51401 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562517054
##########
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##########
@@ -977,12 +978,17 @@ public static double getHotValueThreshold() {
// 100MB
public long maxScanQueueMemByte = 2147483648L / 20;
- @VariableMgr.VarAttr(name = NUM_SCANNER_THREADS, needForward = true, description = {
+ @VariableMgr.VarAttr(name = MAX_SCANNERS_CONCURRENCY, needForward = true, description = {
Review Comment:
max_local_scanners_concurrency vs max_remote_scanners_concurrency ？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3581070922
<details>
<summary>TPC-DS: <b>Total hot run time: 184376 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0, data reload: false
query1 1058 406 389 389
query2 6578 1572 1566 1566
query3 6755 224 218 218
query4 25300 23225 22653 22653
query5 4963 622 464 464
query6 341 238 242 238
query7 4648 513 302 302
query8 325 259 241 241
query9 8728 2556 2576 2556
query10 539 334 301 301
query11 15232 15050 14994 14994
query12 191 123 117 117
query13 1706 587 462 462
query14 11121 8858 8747 8747
query15 214 199 187 187
query16 7688 719 526 526
query17 1212 774 651 651
query18 2048 435 346 346
query19 214 216 196 196
query20 132 127 120 120
query21 217 141 115 115
query22 3776 3957 3836 3836
query23 32835 31860 32029 31860
query24 8473 2431 2408 2408
query25 653 562 495 495
query26 1249 280 172 172
query27 2714 496 355 355
query28 4308 2127 2098 2098
query29 837 658 518 518
query30 311 238 209 209
query31 845 706 637 637
query32 92 82 76 76
query33 612 396 355 355
query34 805 915 550 550
query35 811 824 739 739
query36 887 913 833 833
query37 130 117 95 95
query38 3350 3361 3326 3326
query39 1480 1454 1458 1454
query40 227 129 126 126
query41 67 60 61 60
query42 123 111 108 108
query43 437 443 423 423
query44 1277 765 755 755
query45 201 191 188 188
query46 875 991 635 635
query47 1704 1738 1637 1637
query48 399 415 329 329
query49 774 495 411 411
query50 645 674 402 402
query51 3881 3905 3868 3868
query52 111 108 110 108
query53 232 259 208 208
query54 305 290 279 279
query55 97 94 91 91
query56 338 339 344 339
query57 1140 1170 1115 1115
query58 292 269 273 269
query59 2320 2445 2382 2382
query60 349 350 349 349
query61 160 161 160 160
query62 769 700 663 663
query63 230 201 193 193
query64 4512 1230 900 900
query65 4088 3923 3938 3923
query66 1098 441 348 348
query67 15467 15069 14932 14932
query68 8317 961 629 629
query69 515 354 306 306
query70 1280 1222 1228 1222
query71 490 329 314 314
query72 5837 4773 4844 4773
query73 674 577 341 341
query74 8825 8805 8565 8565
query75 3978 3307 2792 2792
query76 3815 1115 690 690
query77 808 402 308 308
query78 9463 9651 8872 8872
query79 2046 842 597 597
query80 643 600 493 493
query81 492 272 233 233
query82 411 163 136 136
query83 271 259 252 252
query84 267 115 106 106
query85 914 484 445 445
query86 349 318 300 300
query87 3466 3443 3412 3412
query88 3794 2252 2269 2252
query89 389 327 298 298
query90 1983 220 209 209
query91 174 170 143 143
query92 85 75 65 65
query93 1414 991 659 659
query94 726 471 346 346
query95 499 416 403 403
query96 505 547 289 289
query97 2913 2938 2843 2843
query98 237 211 213 211
query99 1394 1376 1266 1266
Total cold run time: 272470 ms
Total hot run time: 184376 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562540277
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -72,6 +72,38 @@ bool ScanLocalState<Derived>::should_run_serial() const {
return _parent->cast<typename Derived::Parent>()._should_run_serial;
}
+int ScanLocalStateBase::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified max_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(48, CpuInfo::num_cores() * 2))
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_scanners_concurrency() ? state->max_scanners_concurrency()
+ : std::max(48, CpuInfo::num_cores() * 2) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int ScanLocalStateBase::min_scanners_concurrency(RuntimeState* state) const {
+ /*
+ * The min concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified min_scan_concurrency_of_scan_scheduler which is set through session variable.
+ * 2. Default: 2 * state->get_query_ctx()->get_scan_scheduler()->get_max_threads()
+ */
+ return state->min_scan_concurrency_of_scan_scheduler()
+ ? state->min_scan_concurrency_of_scan_scheduler()
+ : 2 * state->get_query_ctx()->get_scan_scheduler()->get_max_threads();
Review Comment:
跟cpu 关联，不要跟threads 关联
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Thearas (via GitHub)" <gi...@apache.org>.
Thearas commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3539815989
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584264916
<details>
<summary>TPC-DS: <b>Total hot run time: 184571 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b87eaf23bd13dcfd21756e0823e75065d36d475b, data reload: false
query1 1110 427 392 392
query2 6578 1577 1615 1577
query3 6750 222 226 222
query4 25674 23362 22880 22880
query5 4902 625 483 483
query6 343 249 226 226
query7 4642 510 309 309
query8 302 252 240 240
query9 8708 2577 2572 2572
query10 545 373 302 302
query11 15159 15203 14977 14977
query12 180 117 115 115
query13 1707 571 484 484
query14 11073 8907 8749 8749
query15 212 203 194 194
query16 7526 703 541 541
query17 1220 793 655 655
query18 2013 416 322 322
query19 207 198 171 171
query20 133 124 119 119
query21 215 147 115 115
query22 3996 3951 3877 3877
query23 33039 31988 31758 31758
query24 8410 2383 2387 2383
query25 604 518 444 444
query26 1243 283 168 168
query27 2736 487 339 339
query28 4339 2133 2086 2086
query29 831 639 491 491
query30 314 240 207 207
query31 852 732 589 589
query32 81 74 73 73
query33 600 377 335 335
query34 800 874 546 546
query35 802 855 763 763
query36 896 942 851 851
query37 122 111 92 92
query38 3356 3326 3284 3284
query39 1461 1384 1403 1384
query40 234 132 117 117
query41 65 64 63 63
query42 124 118 110 110
query43 429 460 420 420
query44 1293 759 755 755
query45 201 188 184 184
query46 874 997 645 645
query47 1692 1703 1631 1631
query48 400 423 328 328
query49 781 503 428 428
query50 659 677 410 410
query51 3885 3926 3838 3838
query52 112 113 99 99
query53 243 263 189 189
query54 309 291 278 278
query55 98 96 89 89
query56 336 335 331 331
query57 1156 1148 1090 1090
query58 295 275 269 269
query59 2389 2364 2236 2236
query60 346 344 339 339
query61 166 164 167 164
query62 799 707 681 681
query63 227 195 195 195
query64 4561 1306 1014 1014
query65 4085 3962 4023 3962
query66 1142 466 363 363
query67 15486 14995 14845 14845
query68 7399 952 628 628
query69 525 369 315 315
query70 1287 1203 1210 1203
query71 442 347 328 328
query72 6036 4863 4815 4815
query73 659 573 341 341
query74 8815 8753 8683 8683
query75 3329 3342 2865 2865
query76 3329 1126 699 699
query77 509 405 330 330
query78 9476 9642 8940 8940
query79 2276 803 593 593
query80 634 580 531 531
query81 520 272 249 249
query82 431 160 131 131
query83 260 262 244 244
query84 256 107 97 97
query85 924 487 452 452
query86 382 296 282 282
query87 3485 3474 3463 3463
query88 4125 2237 2250 2237
query89 397 330 297 297
query90 1931 222 216 216
query91 174 171 139 139
query92 89 74 69 69
query93 1975 987 663 663
query94 790 470 366 366
query95 496 404 408 404
query96 509 565 300 300
query97 2893 2981 2842 2842
query98 244 221 210 210
query99 1352 1419 1251 1251
Total cold run time: 272017 ms
Total hot run time: 184571 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584057703
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3540109421
<details>
<summary>TPC-DS: <b>Total hot run time: 190710 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e0bd88b8407116b6be9c54135c09920f79b3756, data reload: false
query1 1011 400 422 400
query2 6573 1761 1739 1739
query3 6758 241 229 229
query4 26544 23654 23467 23467
query5 4456 660 551 551
query6 331 256 238 238
query7 4657 489 297 297
query8 301 264 254 254
query9 8704 2946 2868 2868
query10 513 376 314 314
query11 15546 15061 14854 14854
query12 191 124 121 121
query13 1697 592 533 533
query14 10560 9604 9639 9604
query15 243 188 178 178
query16 7903 693 500 500
query17 1660 844 645 645
query18 2053 436 343 343
query19 250 220 200 200
query20 137 133 125 125
query21 216 139 116 116
query22 3963 4123 4126 4123
query23 34124 33303 32748 32748
query24 8083 2426 2403 2403
query25 623 575 490 490
query26 1230 279 177 177
query27 2690 501 374 374
query28 4362 2338 2309 2309
query29 826 673 544 544
query30 306 225 200 200
query31 887 831 769 769
query32 95 89 84 84
query33 589 409 370 370
query34 773 862 572 572
query35 850 883 769 769
query36 988 1013 909 909
query37 134 123 94 94
query38 3602 3581 3594 3581
query39 1501 1439 1414 1414
query40 239 138 148 138
query41 65 59 61 59
query42 140 135 123 123
query43 492 501 458 458
query44 1292 806 814 806
query45 191 184 178 178
query46 879 991 649 649
query47 1759 1787 1713 1713
query48 413 424 338 338
query49 772 487 437 437
query50 662 689 418 418
query51 3952 4022 3930 3930
query52 127 123 117 117
query53 246 274 201 201
query54 330 315 324 315
query55 92 96 92 92
query56 347 370 359 359
query57 1172 1178 1125 1125
query58 322 297 342 297
query59 2898 2960 2925 2925
query60 374 382 351 351
query61 164 159 152 152
query62 764 704 660 660
query63 249 214 200 200
query64 4443 1164 883 883
query65 4079 3967 3953 3953
query66 1081 442 353 353
query67 15596 15305 14876 14876
query68 8296 982 660 660
query69 488 349 311 311
query70 1268 1179 1204 1179
query71 478 376 350 350
query72 6002 5144 5383 5144
query73 676 750 383 383
query74 9068 8836 8748 8748
query75 3904 3288 2820 2820
query76 3749 1172 791 791
query77 821 448 346 346
query78 9706 9743 8980 8980
query79 3014 854 638 638
query80 760 611 536 536
query81 513 270 227 227
query82 373 171 137 137
query83 301 268 250 250
query84 302 115 102 102
query85 917 479 439 439
query86 390 296 284 284
query87 3786 3762 3693 3693
query88 3608 2313 2284 2284
query89 400 340 302 302
query90 1899 255 235 235
query91 183 164 137 137
query92 80 77 72 72
query93 2078 1001 685 685
query94 688 436 333 333
query95 404 335 344 335
query96 495 593 289 289
query97 2971 2969 2898 2898
query98 266 225 230 225
query99 1505 1411 1285 1285
Total cold run time: 278459 ms
Total hot run time: 190710 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3542351576
<details>
<summary>TPC-DS: <b>Total hot run time: 189875 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22249e2f89b2266965806f7444efdeb3e16ab171, data reload: false
query1 1054 406 426 406
query2 6543 1727 1749 1727
query3 6755 227 220 220
query4 26577 23441 23226 23226
query5 4410 658 531 531
query6 330 243 234 234
query7 4647 485 299 299
query8 305 269 242 242
query9 8674 2910 2878 2878
query10 517 363 317 317
query11 15621 15136 14834 14834
query12 181 125 116 116
query13 1664 580 462 462
query14 10564 9108 9167 9108
query15 198 185 173 173
query16 7797 640 505 505
query17 1376 742 601 601
query18 2039 431 315 315
query19 244 222 237 222
query20 135 133 123 123
query21 211 130 118 118
query22 4186 4422 4280 4280
query23 34118 32974 32894 32894
query24 8050 2261 2379 2261
query25 588 532 465 465
query26 1241 279 160 160
query27 2720 496 361 361
query28 4414 2303 2279 2279
query29 786 626 537 537
query30 301 227 199 199
query31 881 787 711 711
query32 94 80 80 80
query33 566 384 358 358
query34 763 851 527 527
query35 830 858 756 756
query36 983 1023 881 881
query37 124 119 99 99
query38 3582 3486 3507 3486
query39 1472 1421 1417 1417
query40 230 135 124 124
query41 64 61 60 60
query42 138 121 118 118
query43 474 496 447 447
query44 1281 806 799 799
query45 192 195 185 185
query46 880 966 661 661
query47 1804 1853 1776 1776
query48 413 431 355 355
query49 793 530 450 450
query50 655 676 411 411
query51 3890 4020 3912 3912
query52 124 119 110 110
query53 237 276 207 207
query54 324 312 304 304
query55 91 94 88 88
query56 337 351 378 351
query57 1169 1193 1126 1126
query58 315 305 304 304
query59 2863 2957 2909 2909
query60 394 393 363 363
query61 193 185 195 185
query62 766 740 687 687
query63 236 202 199 199
query64 4592 1314 988 988
query65 4033 3917 3943 3917
query66 1093 455 373 373
query67 15446 15152 15088 15088
query68 8114 949 662 662
query69 509 347 314 314
query70 1277 1236 1229 1229
query71 440 378 356 356
query72 6395 5123 5315 5123
query73 657 760 370 370
query74 8899 9124 8779 8779
query75 3299 3262 2857 2857
query76 3288 1119 748 748
query77 548 411 349 349
query78 9755 9852 8978 8978
query79 2845 820 612 612
query80 680 602 526 526
query81 503 263 228 228
query82 438 169 139 139
query83 298 271 248 248
query84 306 111 101 101
query85 927 484 444 444
query86 378 303 294 294
query87 3694 3702 3622 3622
query88 3674 2278 2275 2275
query89 396 345 303 303
query90 2000 236 223 223
query91 169 164 137 137
query92 85 78 71 71
query93 2101 954 684 684
query94 726 432 327 327
query95 403 345 332 332
query96 482 593 295 295
query97 2941 2953 2854 2854
query98 261 220 218 218
query99 1446 1385 1272 1272
Total cold run time: 276502 ms
Total hot run time: 189875 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3569868773
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6d4a5a626312a4f4c9a77a0ad5c412dca03620b0_6d4a5a626312a4f4c9a77a0ad5c412dca03620b0_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6d4a5a626312a4f4c9a77a0ad5c412dca03620b0_6d4a5a626312a4f4c9a77a0ad5c412dca03620b0_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.86% (1707/2111) |
| Line Coverage | 66.85% (29887/44708) |
| Region Coverage | 67.38% (14900/22114) |
| Branch Coverage | 57.57% (7926/13768) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584674096
# FE Regression Coverage Report
Increment line coverage ` 0.00% (0/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_b87eaf23bd13dcfd21756e0823e75065d36d475b_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_b87eaf23bd13dcfd21756e0823e75065d36d475b_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584901203
skip check_coverage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3579869432
<details>
<summary>TPC-H: <b>Total hot run time: 34726 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2462a11bc42619b4fdcffd67b314243593f9dc64, data reload: false
------ Round 1 ----------------------------------
q1 17610 5507 5115 5115
q2 2041 315 215 215
q3 10263 1243 729 729
q4 10235 920 374 374
q5 7467 2499 2100 2100
q6 185 176 138 138
q7 944 784 655 655
q8 9339 1416 1230 1230
q9 7319 5473 5550 5473
q10 6883 2258 1782 1782
q11 496 320 294 294
q12 319 417 226 226
q13 17785 3650 3128 3128
q14 236 253 224 224
q15 584 522 508 508
q16 1039 1022 965 965
q17 599 871 343 343
q18 7691 7248 7193 7193
q19 943 955 545 545
q20 348 344 230 230
q21 4000 3179 2315 2315
q22 1069 998 944 944
Total cold run time: 107395 ms
Total hot run time: 34726 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5149 5178 5162 5162
q2 316 401 316 316
q3 2249 2687 2318 2318
q4 1355 1838 1358 1358
q5 4456 5277 5002 5002
q6 183 170 135 135
q7 2143 1997 1856 1856
q8 2643 2808 2764 2764
q9 8059 7520 7641 7520
q10 3057 3354 2811 2811
q11 618 525 504 504
q12 774 798 616 616
q13 3657 3967 3219 3219
q14 302 324 323 323
q15 581 513 510 510
q16 1121 1168 1105 1105
q17 1291 1791 1262 1262
q18 8030 7760 7710 7710
q19 841 916 1149 916
q20 2041 2046 1975 1975
q21 5126 4681 4489 4489
q22 1113 1052 1000 1000
Total cold run time: 55105 ms
Total hot run time: 52871 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2563740017
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -31,6 +31,39 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
+
+int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of file scanners for each FileScanLocalState is determined by:
+ * 1. User specified max_file_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(512, CpuInfo::num_cores() * 10)
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_file_scanners_concurrency() ? state->max_file_scanners_concurrency()
+ : std::max(512, CpuInfo::num_cores() * 10) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
Review Comment:
这里为什么要 乘 state->query_parallel_instance_num()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3578778228
# FE UT Coverage Report
Increment line coverage ` 66.67% (4/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_98f7f81cbd7268940b4d1428e678607919e50b8c/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_98f7f81cbd7268940b4d1428e678607919e50b8c/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584061178
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3570241802
# FE UT Coverage Report
Increment line coverage ` 66.67% (4/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_6d4a5a626312a4f4c9a77a0ad5c412dca03620b0/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_6d4a5a626312a4f4c9a77a0ad5c412dca03620b0/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3578899786
<details>
<summary>TPC-H: <b>Total hot run time: 35292 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 98f7f81cbd7268940b4d1428e678607919e50b8c, data reload: false
------ Round 1 ----------------------------------
q1 17621 5507 5097 5097
q2 2019 330 220 220
q3 10241 1239 733 733
q4 10230 927 368 368
q5 7499 2414 2134 2134
q6 197 176 139 139
q7 944 801 653 653
q8 9378 1460 1339 1339
q9 7507 5506 5513 5506
q10 6898 2249 1794 1794
q11 504 303 294 294
q12 328 363 222 222
q13 17807 3697 3039 3039
q14 238 235 215 215
q15 595 517 506 506
q16 1030 1041 966 966
q17 625 896 382 382
q18 7654 7382 7042 7042
q19 1103 948 548 548
q20 354 352 230 230
q21 3709 3113 2898 2898
q22 1053 977 967 967
Total cold run time: 107534 ms
Total hot run time: 35292 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5144 5111 5135 5111
q2 339 390 297 297
q3 2278 2702 2316 2316
q4 1330 1877 1367 1367
q5 4470 5152 4973 4973
q6 181 181 155 155
q7 2137 2009 1808 1808
q8 2649 2876 2751 2751
q9 7608 7528 7486 7486
q10 3122 3305 2827 2827
q11 629 530 491 491
q12 699 806 638 638
q13 3644 4004 3508 3508
q14 277 304 281 281
q15 549 501 535 501
q16 1095 1147 1037 1037
q17 1259 1807 1284 1284
q18 7878 7777 7687 7687
q19 811 1194 1056 1056
q20 1960 1973 1829 1829
q21 4806 4461 4357 4357
q22 1122 1060 1002 1002
Total cold run time: 53987 ms
Total hot run time: 52762 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3579491236
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2462a11bc42619b4fdcffd67b314243593f9dc64_2462a11bc42619b4fdcffd67b314243593f9dc64_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2462a11bc42619b4fdcffd67b314243593f9dc64_2462a11bc42619b4fdcffd67b314243593f9dc64_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.86% (1707/2111) |
| Line Coverage | 66.88% (29899/44708) |
| Region Coverage | 67.42% (14910/22114) |
| Branch Coverage | 57.58% (7928/13768) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3579446287
# FE Regression Coverage Report
Increment line coverage ` 66.67% (4/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_98f7f81cbd7268940b4d1428e678607919e50b8c_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_98f7f81cbd7268940b4d1428e678607919e50b8c_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584273806
<details>
<summary>ClickBench: <b>Total hot run time: 27.16 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b87eaf23bd13dcfd21756e0823e75065d36d475b, data reload: false
query1 0.05 0.05 0.04
query2 0.12 0.05 0.04
query3 0.26 0.08 0.09
query4 1.60 0.12 0.11
query5 0.28 0.25 0.25
query6 1.16 0.65 0.64
query7 0.03 0.03 0.03
query8 0.06 0.04 0.04
query9 0.59 0.50 0.51
query10 0.56 0.58 0.56
query11 0.17 0.11 0.10
query12 0.16 0.12 0.11
query13 0.62 0.60 0.61
query14 0.98 1.01 1.00
query15 0.82 0.80 0.81
query16 0.41 0.41 0.42
query17 1.07 1.04 1.00
query18 0.23 0.22 0.22
query19 1.94 1.73 1.73
query20 0.02 0.02 0.02
query21 15.45 0.26 0.14
query22 4.94 0.05 0.05
query23 15.99 0.27 0.10
query24 1.21 0.66 0.18
query25 0.08 0.11 0.05
query26 0.14 0.13 0.13
query27 0.05 0.04 0.06
query28 4.07 1.21 1.02
query29 12.56 3.90 3.21
query30 0.29 0.13 0.12
query31 2.82 0.62 0.40
query32 3.24 0.54 0.46
query33 3.07 3.06 3.10
query34 16.86 5.20 4.52
query35 4.58 4.56 4.60
query36 0.67 0.50 0.49
query37 0.10 0.07 0.06
query38 0.07 0.04 0.04
query39 0.05 0.03 0.03
query40 0.17 0.13 0.13
query41 0.08 0.04 0.03
query42 0.04 0.03 0.03
query43 0.05 0.03 0.04
Total cold run time: 97.71 s
Total hot run time: 27.16 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3541658527
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3541792570
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/22249e2f89b2266965806f7444efdeb3e16ab171_22249e2f89b2266965806f7444efdeb3e16ab171_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/22249e2f89b2266965806f7444efdeb3e16ab171_22249e2f89b2266965806f7444efdeb3e16ab171_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.56% (1683/2089) |
| Line Coverage | 66.68% (29684/44518) |
| Region Coverage | 67.22% (14808/22028) |
| Branch Coverage | 57.52% (7884/13706) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3542033407
# FE UT Coverage Report
Increment line coverage ` 100.00% (2/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_22249e2f89b2266965806f7444efdeb3e16ab171/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_22249e2f89b2266965806f7444efdeb3e16ab171/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3540526231
# FE Regression Coverage Report
Increment line coverage ` 100.00% (2/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_2e0bd88b8407116b6be9c54135c09920f79b3756_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_2e0bd88b8407116b6be9c54135c09920f79b3756_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3544940463
# FE Regression Coverage Report
Increment line coverage ` 100.00% (2/2)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_22249e2f89b2266965806f7444efdeb3e16ab171_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_22249e2f89b2266965806f7444efdeb3e16ab171_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3570396205
<details>
<summary>TPC-DS: <b>Total hot run time: 187790 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6d4a5a626312a4f4c9a77a0ad5c412dca03620b0, data reload: false
query1 1001 423 387 387
query2 6555 1679 1666 1666
query3 6752 215 227 215
query4 25758 23675 23446 23446
query5 4929 653 487 487
query6 346 235 220 220
query7 4671 495 317 317
query8 295 248 249 248
query9 8719 2608 2640 2608
query10 513 347 287 287
query11 15625 15152 15184 15152
query12 186 122 115 115
query13 1690 570 464 464
query14 11146 9054 9009 9009
query15 203 198 172 172
query16 7597 721 504 504
query17 1180 749 618 618
query18 2020 426 319 319
query19 204 193 168 168
query20 135 127 123 123
query21 220 131 113 113
query22 4087 4322 3986 3986
query23 33936 32990 32812 32812
query24 8474 2389 2364 2364
query25 634 523 460 460
query26 1236 275 166 166
query27 2734 497 357 357
query28 4348 2187 2174 2174
query29 760 668 483 483
query30 301 221 196 196
query31 918 803 724 724
query32 82 76 73 73
query33 580 358 324 324
query34 804 856 524 524
query35 802 808 716 716
query36 910 948 897 897
query37 114 114 93 93
query38 3478 3521 3440 3440
query39 1451 1417 1394 1394
query40 225 131 128 128
query41 71 62 65 62
query42 126 112 107 107
query43 466 485 450 450
query44 1242 773 772 772
query45 186 180 171 171
query46 869 981 648 648
query47 1778 1817 1724 1724
query48 419 410 323 323
query49 777 482 398 398
query50 652 678 410 410
query51 3973 3911 3859 3859
query52 114 113 108 108
query53 244 259 198 198
query54 302 291 273 273
query55 89 91 83 83
query56 321 325 313 313
query57 1167 1187 1110 1110
query58 292 273 275 273
query59 2474 2551 2506 2506
query60 342 346 348 346
query61 160 164 167 164
query62 795 730 656 656
query63 233 195 193 193
query64 4414 1185 937 937
query65 4030 3935 3924 3924
query66 1139 454 352 352
query67 15114 15013 14881 14881
query68 4607 963 650 650
query69 503 341 304 304
query70 1271 1195 1265 1195
query71 419 328 312 312
query72 6042 5027 4974 4974
query73 662 565 354 354
query74 9295 9071 9043 9043
query75 3279 3303 2812 2812
query76 3307 1129 707 707
query77 499 409 329 329
query78 9557 9730 8828 8828
query79 1894 865 603 603
query80 723 583 525 525
query81 518 258 225 225
query82 240 166 130 130
query83 278 275 250 250
query84 265 105 89 89
query85 896 488 446 446
query86 384 308 292 292
query87 3669 3711 3572 3572
query88 2783 2274 2243 2243
query89 383 338 294 294
query90 1826 223 221 221
query91 195 163 147 147
query92 79 68 69 68
query93 1236 1009 690 690
query94 638 446 339 339
query95 418 321 323 321
query96 491 550 290 290
query97 2858 2950 2871 2871
query98 246 214 204 204
query99 1284 1405 1272 1272
Total cold run time: 268153 ms
Total hot run time: 187790 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3579366961
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3579917789
<details>
<summary>TPC-DS: <b>Total hot run time: 185569 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2462a11bc42619b4fdcffd67b314243593f9dc64, data reload: false
query1 1040 411 396 396
query2 6585 1694 1699 1694
query3 6762 241 231 231
query4 25815 22632 22811 22632
query5 5789 655 480 480
query6 347 279 250 250
query7 4654 492 306 306
query8 308 276 264 264
query9 8741 2574 2561 2561
query10 576 383 327 327
query11 15298 14789 14605 14605
query12 187 123 117 117
query13 1725 616 476 476
query14 11923 8983 8920 8920
query15 265 203 187 187
query16 7853 695 541 541
query17 1638 777 624 624
query18 2083 420 337 337
query19 309 228 192 192
query20 135 127 123 123
query21 218 138 114 114
query22 3927 4051 4044 4044
query23 33152 32373 32134 32134
query24 8293 2379 2376 2376
query25 574 523 443 443
query26 1218 280 168 168
query27 2686 490 364 364
query28 4341 2144 2119 2119
query29 789 649 521 521
query30 306 247 221 221
query31 813 733 638 638
query32 85 71 69 69
query33 617 384 339 339
query34 756 878 533 533
query35 795 842 762 762
query36 947 986 869 869
query37 130 109 88 88
query38 3354 3387 3305 3305
query39 1459 1404 1422 1404
query40 225 127 121 121
query41 67 61 60 60
query42 133 125 121 121
query43 437 447 410 410
query44 1278 758 760 758
query45 200 195 183 183
query46 860 985 626 626
query47 1732 1749 1678 1678
query48 438 425 342 342
query49 808 500 421 421
query50 665 674 416 416
query51 3870 3889 3866 3866
query52 114 112 118 112
query53 240 264 202 202
query54 328 303 287 287
query55 96 101 105 101
query56 364 352 343 343
query57 1160 1186 1109 1109
query58 301 274 280 274
query59 2763 2908 2768 2768
query60 351 354 336 336
query61 165 165 162 162
query62 766 738 651 651
query63 236 194 203 194
query64 4449 1188 912 912
query65 4068 3950 4030 3950
query66 1049 454 331 331
query67 15188 14969 14743 14743
query68 8272 958 649 649
query69 547 355 306 306
query70 1219 1190 1182 1182
query71 487 349 327 327
query72 5768 5219 5531 5219
query73 642 770 371 371
query74 8899 8490 8569 8490
query75 3582 3347 2852 2852
query76 3447 1126 692 692
query77 750 410 310 310
query78 9697 9720 8985 8985
query79 2581 852 592 592
query80 734 597 502 502
query81 496 273 240 240
query82 475 172 133 133
query83 305 262 256 256
query84 304 129 100 100
query85 907 496 441 441
query86 341 278 273 273
query87 3558 3466 3400 3400
query88 3679 2265 2277 2265
query89 412 339 314 314
query90 1977 221 209 209
query91 175 172 142 142
query92 84 71 65 65
query93 1637 950 667 667
query94 752 445 329 329
query95 477 412 397 397
query96 507 558 289 289
query97 2918 2952 2872 2872
query98 242 216 214 214
query99 1427 1387 1235 1235
Total cold run time: 276142 ms
Total hot run time: 185569 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3579944771
<details>
<summary>ClickBench: <b>Total hot run time: 28.41 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2462a11bc42619b4fdcffd67b314243593f9dc64, data reload: false
query1 0.07 0.06 0.05
query2 0.15 0.07 0.07
query3 0.33 0.10 0.09
query4 1.61 0.11 0.11
query5 0.28 0.26 0.27
query6 1.19 0.66 0.67
query7 0.03 0.03 0.03
query8 0.07 0.06 0.06
query9 0.60 0.54 0.51
query10 0.59 0.58 0.58
query11 0.24 0.12 0.12
query12 0.24 0.13 0.13
query13 0.64 0.61 0.60
query14 1.00 0.98 1.00
query15 0.92 0.82 0.82
query16 0.41 0.40 0.40
query17 1.10 1.09 1.03
query18 0.10 0.09 0.08
query19 2.01 1.90 1.92
query20 0.01 0.01 0.02
query21 15.44 0.27 0.23
query22 5.53 0.09 0.10
query23 15.80 0.36 0.21
query24 1.07 0.39 0.27
query25 0.10 0.09 0.09
query26 0.19 0.18 0.18
query27 0.09 0.10 0.10
query28 3.37 1.29 1.10
query29 12.56 3.89 3.19
query30 0.33 0.13 0.11
query31 2.80 0.61 0.43
query32 3.24 0.57 0.48
query33 3.05 3.15 3.31
query34 15.80 5.25 4.64
query35 4.62 4.66 4.69
query36 0.60 0.49 0.49
query37 0.23 0.08 0.08
query38 0.18 0.06 0.06
query39 0.06 0.05 0.05
query40 0.17 0.16 0.15
query41 0.10 0.06 0.06
query42 0.09 0.05 0.05
query43 0.07 0.05 0.05
Total cold run time: 97.08 s
Total hot run time: 28.41 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3580550803
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584006257
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584184783
# FE UT Coverage Report
Increment line coverage ` 80.00% (8/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_b87eaf23bd13dcfd21756e0823e75065d36d475b/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_b87eaf23bd13dcfd21756e0823e75065d36d475b/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3575703703
<details>
<summary>TPC-DS: <b>Total hot run time: 185792 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f1aaee068839444ae7dd524bf84bcce15c4fba9, data reload: false
query1 1047 411 422 411
query2 6583 1718 1731 1718
query3 6759 232 224 224
query4 26269 22556 22644 22556
query5 4435 650 476 476
query6 344 268 265 265
query7 4641 486 302 302
query8 300 253 248 248
query9 8692 2574 2617 2574
query10 537 362 316 316
query11 15105 14970 14651 14651
query12 177 115 110 110
query13 1684 605 496 496
query14 10792 9074 9077 9074
query15 214 205 193 193
query16 7843 709 525 525
query17 1665 788 603 603
query18 2050 428 316 316
query19 252 217 189 189
query20 126 118 120 118
query21 223 135 115 115
query22 3927 4078 3869 3869
query23 33235 32124 32391 32124
query24 8464 2461 2399 2399
query25 559 513 448 448
query26 1226 276 163 163
query27 2709 488 374 374
query28 4342 2140 2132 2132
query29 775 651 505 505
query30 309 250 214 214
query31 840 727 634 634
query32 81 74 65 65
query33 590 375 331 331
query34 750 884 527 527
query35 783 850 750 750
query36 934 1000 866 866
query37 125 108 89 89
query38 3315 3351 3317 3317
query39 1457 1418 1434 1418
query40 223 126 116 116
query41 66 61 61 61
query42 125 114 111 111
query43 451 494 444 444
query44 1229 762 753 753
query45 202 190 190 190
query46 889 991 636 636
query47 1736 1782 1682 1682
query48 421 432 353 353
query49 772 496 414 414
query50 659 673 412 412
query51 3980 4031 3899 3899
query52 106 110 104 104
query53 250 263 193 193
query54 322 297 275 275
query55 87 81 87 81
query56 343 346 315 315
query57 1166 1176 1097 1097
query58 297 295 275 275
query59 2735 2984 2766 2766
query60 362 349 343 343
query61 165 187 165 165
query62 751 720 665 665
query63 235 198 194 194
query64 4417 1209 883 883
query65 4074 4019 3994 3994
query66 1092 428 335 335
query67 15299 14914 14866 14866
query68 5178 922 656 656
query69 524 351 294 294
query70 1274 1244 1283 1244
query71 423 337 315 315
query72 5816 5311 5528 5311
query73 642 730 387 387
query74 8606 8681 8410 8410
query75 3364 3344 2841 2841
query76 3313 1150 727 727
query77 517 393 335 335
query78 9666 9770 8961 8961
query79 2794 846 629 629
query80 820 606 581 581
query81 532 278 231 231
query82 706 168 129 129
query83 303 264 249 249
query84 310 113 105 105
query85 920 489 440 440
query86 397 282 285 282
query87 3527 3509 3421 3421
query88 3786 2216 2218 2216
query89 386 334 302 302
query90 1861 223 216 216
query91 174 170 145 145
query92 81 71 61 61
query93 2574 938 659 659
query94 763 424 328 328
query95 479 397 404 397
query96 512 589 309 309
query97 2905 2960 2839 2839
query98 226 213 203 203
query99 1441 1380 1242 1242
Total cold run time: 271433 ms
Total hot run time: 185792 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3542373223
<details>
<summary>ClickBench: <b>Total hot run time: 28.32 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22249e2f89b2266965806f7444efdeb3e16ab171, data reload: false
query1 0.05 0.04 0.05
query2 0.13 0.07 0.06
query3 0.32 0.07 0.08
query4 1.60 0.08 0.09
query5 0.26 0.26 0.27
query6 1.18 0.64 0.64
query7 0.03 0.03 0.03
query8 0.08 0.06 0.07
query9 0.58 0.51 0.51
query10 0.58 0.55 0.56
query11 0.24 0.13 0.14
query12 0.26 0.15 0.15
query13 0.63 0.62 0.62
query14 1.00 0.98 0.99
query15 0.85 0.81 0.80
query16 0.40 0.38 0.39
query17 1.04 1.04 1.10
query18 0.23 0.22 0.22
query19 1.87 1.81 1.79
query20 0.02 0.02 0.01
query21 15.40 0.27 0.24
query22 4.98 0.10 0.09
query23 15.38 0.37 0.22
query24 2.18 0.46 0.35
query25 0.09 0.09 0.09
query26 0.18 0.18 0.17
query27 0.09 0.09 0.08
query28 4.01 1.26 1.08
query29 12.54 3.96 3.26
query30 0.35 0.12 0.12
query31 2.82 0.63 0.42
query32 3.23 0.58 0.50
query33 3.03 3.12 3.08
query34 16.27 5.18 4.59
query35 4.58 4.62 4.62
query36 0.61 0.51 0.48
query37 0.24 0.09 0.08
query38 0.19 0.06 0.06
query39 0.06 0.05 0.04
query40 0.19 0.18 0.16
query41 0.12 0.06 0.07
query42 0.07 0.06 0.06
query43 0.07 0.06 0.05
Total cold run time: 98.03 s
Total hot run time: 28.32 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3575232569
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/6f1aaee068839444ae7dd524bf84bcce15c4fba9_6f1aaee068839444ae7dd524bf84bcce15c4fba9_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/6f1aaee068839444ae7dd524bf84bcce15c4fba9_6f1aaee068839444ae7dd524bf84bcce15c4fba9_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.86% (1707/2111) |
| Line Coverage | 66.86% (29891/44708) |
| Region Coverage | 67.41% (14906/22114) |
| Branch Coverage | 57.57% (7926/13768) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3575649266
<details>
<summary>TPC-H: <b>Total hot run time: 36199 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f1aaee068839444ae7dd524bf84bcce15c4fba9, data reload: false
------ Round 1 ----------------------------------
q1 17467 5495 5139 5139
q2 2024 329 211 211
q3 10187 1225 729 729
q4 10235 904 377 377
q5 7540 2470 2156 2156
q6 194 170 136 136
q7 956 783 645 645
q8 9349 1434 1263 1263
q9 7513 5414 5402 5402
q10 6941 2288 1835 1835
q11 501 294 294 294
q12 336 365 225 225
q13 17793 3650 3061 3061
q14 233 235 209 209
q15 589 516 507 507
q16 1035 1011 961 961
q17 611 885 364 364
q18 8116 7647 7691 7647
q19 1116 988 591 591
q20 387 367 246 246
q21 4364 3852 3170 3170
q22 1180 1104 1031 1031
Total cold run time: 108667 ms
Total hot run time: 36199 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5399 5487 5373 5373
q2 329 411 340 340
q3 2514 2824 2492 2492
q4 1455 1887 1441 1441
q5 4585 5087 5005 5005
q6 173 168 126 126
q7 2060 2013 1790 1790
q8 2849 2839 2705 2705
q9 7753 7795 7444 7444
q10 3046 3201 2685 2685
q11 578 505 476 476
q12 621 730 581 581
q13 3278 3674 3095 3095
q14 271 284 259 259
q15 537 502 502 502
q16 1045 1079 1027 1027
q17 1204 1714 1230 1230
q18 7373 7270 7111 7111
q19 798 1152 1022 1022
q20 1947 2003 1836 1836
q21 4859 4550 4391 4391
q22 1139 1073 1028 1028
Total cold run time: 53813 ms
Total hot run time: 51959 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3540083234
<details>
<summary>TPC-H: <b>Total hot run time: 34675 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e0bd88b8407116b6be9c54135c09920f79b3756, data reload: false
------ Round 1 ----------------------------------
q1 17471 5677 5249 5249
q2 2077 313 207 207
q3 10157 1275 714 714
q4 10223 914 383 383
q5 7503 2445 2098 2098
q6 192 178 139 139
q7 936 768 630 630
q8 9350 1386 1278 1278
q9 7419 5404 5391 5391
q10 6925 2272 1782 1782
q11 503 300 285 285
q12 339 378 233 233
q13 17800 3725 3028 3028
q14 233 237 222 222
q15 564 517 520 517
q16 1027 1005 929 929
q17 625 885 354 354
q18 7744 7111 7167 7111
q19 1099 971 583 583
q20 370 361 247 247
q21 3896 2490 2305 2305
q22 1103 1039 990 990
Total cold run time: 107556 ms
Total hot run time: 34675 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5417 5344 5321 5321
q2 277 319 236 236
q3 2247 2724 2352 2352
q4 1392 1806 1380 1380
q5 4364 5138 5032 5032
q6 188 170 135 135
q7 2012 1986 1869 1869
q8 2662 2890 2773 2773
q9 7550 7544 7466 7466
q10 3039 3304 2811 2811
q11 593 531 522 522
q12 682 793 633 633
q13 3549 3996 3347 3347
q14 307 295 302 295
q15 533 501 517 501
q16 1064 1234 1088 1088
q17 1278 1802 1329 1329
q18 7941 7842 7666 7666
q19 837 844 1196 844
q20 1924 1985 1822 1822
q21 4817 4472 4407 4407
q22 1094 1087 1036 1036
Total cold run time: 53767 ms
Total hot run time: 52865 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3539858549
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3544904228
# BE Regression && UT Coverage Report
Increment line coverage `97.22% (35/36)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_22249e2f89b2266965806f7444efdeb3e16ab171_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_22249e2f89b2266965806f7444efdeb3e16ab171_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 71.51% (24347/34046) |
| Line Coverage | 57.98% (253031/436409) |
| Region Coverage | 53.17% (210489/395881) |
| Branch Coverage | 54.57% (89951/164834) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562538354
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -72,6 +72,38 @@ bool ScanLocalState<Derived>::should_run_serial() const {
return _parent->cast<typename Derived::Parent>()._should_run_serial;
}
+int ScanLocalStateBase::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified max_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(48, CpuInfo::num_cores() * 2))
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_scanners_concurrency() ? state->max_scanners_concurrency()
+ : std::max(48, CpuInfo::num_cores() * 2) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int ScanLocalStateBase::min_scanners_concurrency(RuntimeState* state) const {
+ /*
+ * The min concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified min_scan_concurrency_of_scan_scheduler which is set through session variable.
+ * 2. Default: 2 * state->get_query_ctx()->get_scan_scheduler()->get_max_threads()
+ */
+ return state->min_scan_concurrency_of_scan_scheduler()
+ ? state->min_scan_concurrency_of_scan_scheduler()
+ : 2 * state->get_query_ctx()->get_scan_scheduler()->get_max_threads();
+}
+
+vectorized::ScannerScheduler* ScanLocalStateBase::scan_scheduler(RuntimeState* state) const {
+ return state->get_query_ctx()->get_scan_scheduler();
Review Comment:
这个默认使用remote scan scheduler； 然后再olap scan operator 里，返回local scan scheduler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584006210
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584006687
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584236606
<details>
<summary>TPC-H: <b>Total hot run time: 34513 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b87eaf23bd13dcfd21756e0823e75065d36d475b, data reload: false
------ Round 1 ----------------------------------
q1 17405 5135 4954 4954
q2 2060 320 214 214
q3 10147 1299 735 735
q4 10259 881 371 371
q5 7525 2519 2246 2246
q6 180 167 137 137
q7 949 813 635 635
q8 9347 1365 1032 1032
q9 7147 5505 5371 5371
q10 6882 2223 1828 1828
q11 529 322 296 296
q12 358 375 232 232
q13 17769 3668 3044 3044
q14 235 226 210 210
q15 591 521 509 509
q16 1050 1023 954 954
q17 606 773 506 506
q18 8026 7101 7060 7060
q19 1090 961 590 590
q20 343 342 231 231
q21 3564 3696 2396 2396
q22 1003 968 962 962
Total cold run time: 107065 ms
Total hot run time: 34513 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5155 4955 4913 4913
q2 339 397 309 309
q3 2209 2707 2277 2277
q4 1316 1756 1360 1360
q5 4267 4593 4621 4593
q6 224 175 130 130
q7 2077 2009 1823 1823
q8 2706 2516 2473 2473
q9 7546 7504 7473 7473
q10 3082 3342 2814 2814
q11 634 517 493 493
q12 660 735 595 595
q13 3593 3936 3294 3294
q14 297 328 290 290
q15 556 504 512 504
q16 1111 1132 1085 1085
q17 1166 1473 1412 1412
q18 7848 7782 7643 7643
q19 803 818 821 818
q20 2018 2041 1906 1906
q21 4879 4337 4213 4213
q22 1037 1013 981 981
Total cold run time: 53523 ms
Total hot run time: 51399 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3575724065
<details>
<summary>ClickBench: <b>Total hot run time: 28.51 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f1aaee068839444ae7dd524bf84bcce15c4fba9, data reload: false
query1 0.05 0.04 0.04
query2 0.12 0.07 0.06
query3 0.30 0.08 0.07
query4 1.61 0.09 0.09
query5 0.28 0.27 0.27
query6 1.18 0.66 0.65
query7 0.03 0.02 0.03
query8 0.08 0.06 0.07
query9 0.58 0.54 0.52
query10 0.58 0.58 0.58
query11 0.23 0.13 0.13
query12 0.25 0.14 0.15
query13 0.63 0.61 0.62
query14 0.99 0.99 0.99
query15 0.85 0.81 0.80
query16 0.40 0.38 0.39
query17 1.05 1.09 1.02
query18 0.20 0.20 0.19
query19 1.85 1.85 1.89
query20 0.02 0.01 0.02
query21 15.40 0.28 0.24
query22 4.97 0.10 0.10
query23 15.40 0.38 0.22
query24 2.26 0.43 0.33
query25 0.10 0.08 0.08
query26 0.19 0.18 0.18
query27 0.09 0.09 0.09
query28 3.73 1.28 1.09
query29 12.54 3.92 3.22
query30 0.33 0.13 0.11
query31 2.80 0.60 0.43
query32 3.24 0.58 0.50
query33 3.04 3.11 3.11
query34 16.50 5.22 4.65
query35 4.63 4.68 4.66
query36 0.59 0.50 0.50
query37 0.23 0.09 0.09
query38 0.21 0.06 0.06
query39 0.06 0.05 0.05
query40 0.20 0.17 0.16
query41 0.11 0.06 0.06
query42 0.07 0.05 0.05
query43 0.06 0.06 0.05
Total cold run time: 98.03 s
Total hot run time: 28.51 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3581091087
<details>
<summary>ClickBench: <b>Total hot run time: 28.37 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0, data reload: false
query1 0.05 0.04 0.04
query2 0.14 0.07 0.06
query3 0.32 0.07 0.08
query4 1.60 0.10 0.10
query5 0.26 0.25 0.24
query6 1.18 0.65 0.62
query7 0.03 0.02 0.03
query8 0.07 0.07 0.06
query9 0.58 0.52 0.50
query10 0.55 0.56 0.56
query11 0.26 0.14 0.14
query12 0.27 0.14 0.14
query13 0.65 0.63 0.62
query14 1.01 1.01 1.00
query15 0.88 0.81 0.81
query16 0.38 0.38 0.40
query17 1.02 1.05 0.99
query18 0.23 0.22 0.22
query19 1.86 1.77 1.76
query20 0.02 0.01 0.02
query21 15.39 0.28 0.23
query22 4.98 0.10 0.09
query23 15.42 0.37 0.22
query24 2.41 0.43 0.29
query25 0.10 0.09 0.09
query26 0.19 0.17 0.17
query27 0.10 0.09 0.09
query28 3.78 1.36 1.18
query29 12.60 3.98 3.32
query30 0.31 0.13 0.11
query31 2.80 0.61 0.45
query32 3.26 0.57 0.49
query33 3.08 3.09 3.12
query34 16.77 5.14 4.57
query35 4.70 4.70 4.67
query36 0.62 0.49 0.48
query37 0.24 0.09 0.08
query38 0.21 0.06 0.05
query39 0.07 0.05 0.05
query40 0.20 0.17 0.16
query41 0.11 0.06 0.06
query42 0.08 0.05 0.04
query43 0.06 0.05 0.06
Total cold run time: 98.84 s
Total hot run time: 28.37 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2562538984
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -72,6 +72,38 @@ bool ScanLocalState<Derived>::should_run_serial() const {
return _parent->cast<typename Derived::Parent>()._should_run_serial;
}
+int ScanLocalStateBase::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified max_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(48, CpuInfo::num_cores() * 2))
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_scanners_concurrency() ? state->max_scanners_concurrency()
+ : std::max(48, CpuInfo::num_cores() * 2) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int ScanLocalStateBase::min_scanners_concurrency(RuntimeState* state) const {
+ /*
+ * The min concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified min_scan_concurrency_of_scan_scheduler which is set through session variable.
+ * 2. Default: 2 * state->get_query_ctx()->get_scan_scheduler()->get_max_threads()
+ */
+ return state->min_scan_concurrency_of_scan_scheduler()
Review Comment:
min_scan_concurrency_of_scan_scheduler > 0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3578950312
<details>
<summary>ClickBench: <b>Total hot run time: 28.47 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 98f7f81cbd7268940b4d1428e678607919e50b8c, data reload: false
query1 0.05 0.05 0.05
query2 0.13 0.07 0.06
query3 0.31 0.08 0.08
query4 1.60 0.10 0.09
query5 0.28 0.26 0.26
query6 1.18 0.66 0.64
query7 0.03 0.03 0.02
query8 0.12 0.06 0.06
query9 0.59 0.54 0.52
query10 0.57 0.57 0.58
query11 0.23 0.14 0.12
query12 0.25 0.14 0.14
query13 0.64 0.62 0.61
query14 1.00 0.99 0.98
query15 0.85 0.81 0.81
query16 0.40 0.39 0.39
query17 1.07 1.03 1.02
query18 0.20 0.16 0.17
query19 1.91 1.93 1.91
query20 0.02 0.01 0.01
query21 15.39 0.28 0.23
query22 4.97 0.11 0.11
query23 15.39 0.38 0.22
query24 2.31 0.45 0.34
query25 0.09 0.10 0.09
query26 0.19 0.18 0.18
query27 0.09 0.09 0.08
query28 3.92 1.31 1.10
query29 12.54 3.88 3.25
query30 0.32 0.13 0.13
query31 2.80 0.62 0.42
query32 3.24 0.58 0.50
query33 3.06 3.09 3.11
query34 16.68 5.19 4.64
query35 4.62 4.59 4.60
query36 0.59 0.51 0.50
query37 0.23 0.08 0.08
query38 0.19 0.05 0.06
query39 0.06 0.05 0.05
query40 0.20 0.17 0.16
query41 0.11 0.06 0.06
query42 0.07 0.05 0.05
query43 0.06 0.06 0.05
Total cold run time: 98.55 s
Total hot run time: 28.47 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2563711411
##########
be/src/pipeline/exec/file_scan_operator.cpp:
##########
@@ -31,6 +31,39 @@
namespace doris::pipeline {
#include "common/compile_check_begin.h"
+
+int FileScanLocalState::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of file scanners for each FileScanLocalState is determined by:
+ * 1. User specified max_file_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(512, CpuInfo::num_cores() * 10)
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_file_scanners_concurrency() ? state->max_file_scanners_concurrency()
+ : std::max(512, CpuInfo::num_cores() * 10) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int FileScanLocalState::min_scanners_concurrency(RuntimeState* state) const {
Review Comment:
rename to min_scan_concurrency_of_scan_scheduler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #58073:
URL: https://github.com/apache/doris/pull/58073#discussion_r2563750292
##########
be/src/pipeline/exec/scan_operator.cpp:
##########
@@ -72,6 +72,38 @@ bool ScanLocalState<Derived>::should_run_serial() const {
return _parent->cast<typename Derived::Parent>()._should_run_serial;
}
+int ScanLocalStateBase::max_scanners_concurrency(RuntimeState* state) const {
+ // For select * from table limit 10; should just use one thread.
+ if (should_run_serial()) {
+ return 1;
+ }
+ /*
+ * The max concurrency of scanners for each ScanLocalStateBase is determined by:
+ * 1. User specified max_scanners_concurrency which is set through session variable.
+ * 2. Default: std::max(48, CpuInfo::num_cores() * 2))
+ *
+ * If this is a serial operator, the max concurrency should multiply by the number of parallel instances of the operator.
+ */
+ return (state->max_scanners_concurrency() ? state->max_scanners_concurrency()
+ : std::max(48, CpuInfo::num_cores() * 2) * 2) *
+ (state->query_parallel_instance_num() / _parent->parallelism(state));
+}
+
+int ScanLocalStateBase::min_scanners_concurrency(RuntimeState* state) const {
Review Comment:
rename to min_scan_concurrency_of_scan_scheduler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3580440587
# BE UT Coverage Report
Increment line coverage `19.70% (13/66)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/2462a11bc42619b4fdcffd67b314243593f9dc64_2462a11bc42619b4fdcffd67b314243593f9dc64/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/2462a11bc42619b4fdcffd67b314243593f9dc64_2462a11bc42619b4fdcffd67b314243593f9dc64/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.91% (18434/34840) |
| Line Coverage | 38.39% (168468/438813) |
| Region Coverage | 33.16% (130724/394165) |
| Branch Coverage | 34.12% (56282/164948) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3580803158
# FE UT Coverage Report
Increment line coverage ` 80.00% (8/10)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3581024548
<details>
<summary>TPC-H: <b>Total hot run time: 35169 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dd41d8a649b2599efab873ad3bd0f9aa54ca9fc0, data reload: false
------ Round 1 ----------------------------------
q1 17430 4991 5003 4991
q2 2028 320 204 204
q3 10174 1309 746 746
q4 10239 892 374 374
q5 7571 2513 2212 2212
q6 195 173 138 138
q7 968 790 640 640
q8 9353 1369 1010 1010
q9 7382 5276 5380 5276
q10 6885 2250 1797 1797
q11 527 304 291 291
q12 377 369 231 231
q13 17779 3657 3053 3053
q14 236 234 218 218
q15 588 514 514 514
q16 1066 1012 943 943
q17 633 834 438 438
q18 7950 7509 7546 7509
q19 1249 970 638 638
q20 387 363 229 229
q21 4237 4007 2675 2675
q22 1107 1096 1042 1042
Total cold run time: 108361 ms
Total hot run time: 35169 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5123 5123 5139 5123
q2 325 395 337 337
q3 2314 3033 2458 2458
q4 1371 1896 1370 1370
q5 4504 4490 4473 4473
q6 203 164 125 125
q7 1960 1905 1791 1791
q8 2653 2472 2450 2450
q9 7622 7394 7565 7394
q10 3212 3238 2764 2764
q11 567 501 475 475
q12 642 699 555 555
q13 3253 3646 3017 3017
q14 262 284 262 262
q15 539 496 490 490
q16 1032 1043 1009 1009
q17 1098 1357 1390 1357
q18 7270 7140 6925 6925
q19 777 768 748 748
q20 1884 1950 1795 1795
q21 4637 4229 4045 4045
q22 1114 1068 966 966
Total cold run time: 52362 ms
Total hot run time: 49929 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3575400198
# FE UT Coverage Report
Increment line coverage ` 66.67% (4/6)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/58073_6f1aaee068839444ae7dd524bf84bcce15c4fba9/fe_increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/58073_6f1aaee068839444ae7dd524bf84bcce15c4fba9/fe_report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3584108713
# Cloud UT Coverage Report
Increment line coverage ` ` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/b87eaf23bd13dcfd21756e0823e75065d36d475b_b87eaf23bd13dcfd21756e0823e75065d36d475b_cloud/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/b87eaf23bd13dcfd21756e0823e75065d36d475b_b87eaf23bd13dcfd21756e0823e75065d36d475b_cloud/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 80.86% (1707/2111) |
| Line Coverage | 66.84% (29882/44708) |
| Region Coverage | 67.38% (14901/22114) |
| Branch Coverage | 57.59% (7929/13768) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #58073:
URL: https://github.com/apache/doris/pull/58073#issuecomment-3570414432
<details>
<summary>ClickBench: <b>Total hot run time: 28.34 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6d4a5a626312a4f4c9a77a0ad5c412dca03620b0, data reload: false
query1 0.05 0.05 0.04
query2 0.13 0.06 0.06
query3 0.33 0.08 0.08
query4 1.61 0.09 0.09
query5 0.26 0.25 0.24
query6 1.18 0.65 0.63
query7 0.03 0.02 0.03
query8 0.07 0.06 0.06
query9 0.59 0.53 0.51
query10 0.57 0.55 0.55
query11 0.25 0.12 0.14
query12 0.25 0.14 0.14
query13 0.64 0.62 0.61
query14 1.01 0.99 1.01
query15 0.89 0.83 0.82
query16 0.39 0.39 0.40
query17 0.96 1.08 1.05
query18 0.22 0.20 0.21
query19 1.86 1.77 1.80
query20 0.02 0.01 0.01
query21 15.43 0.27 0.23
query22 4.98 0.09 0.09
query23 15.39 0.38 0.22
query24 2.34 0.45 0.31
query25 0.09 0.09 0.10
query26 0.18 0.17 0.18
query27 0.09 0.08 0.08
query28 3.55 1.26 1.09
query29 12.56 3.93 3.32
query30 0.33 0.13 0.12
query31 2.82 0.60 0.44
query32 3.23 0.58 0.50
query33 3.08 3.13 3.02
query34 16.38 5.29 4.63
query35 4.68 4.67 4.69
query36 0.62 0.52 0.48
query37 0.23 0.09 0.09
query38 0.20 0.06 0.06
query39 0.06 0.05 0.04
query40 0.20 0.18 0.16
query41 0.11 0.06 0.06
query42 0.07 0.06 0.05
query43 0.06 0.05 0.04
Total cold run time: 97.99 s
Total hot run time: 28.34 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [Improvement](external) Increase scanner concurrency [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #58073:
URL: https://github.com/apache/doris/pull/58073
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org