You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/11/09 13:32:53 UTC
[PR] [refactor](pipelineX) refine some read dependency [doris]
Mryange opened a new pull request, #26711:
URL: https://github.com/apache/doris/pull/26711
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805343420
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1804845247
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805228071
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805343478
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805310354
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.12 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162729162 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805425883
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 0969fd6dad522aa855c6c0b5a5b592c438999759, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5172 5088 4995 4995
q2 383 195 177 177
q3 2072 2104 2155 2104
q4 1483 1428 1426 1426
q5 4111 4163 4081 4081
q6 256 131 131 131
q7 2065 1617 1603 1603
q8 2796 2746 2750 2746
q9 10424 10357 10214 10214
q10 3505 3554 3551 3551
q11 365 257 254 254
q12 445 298 303 298
q13 4506 4134 4142 4134
q14 330 298 299 298
q15 644 546 571 546
q16 687 625 605 605
q17 1153 1114 1097 1097
q18 7747 7306 7434 7306
q19 1695 1705 1721 1705
q20 584 370 359 359
q21 4929 4560 4581 4560
q22 541 441 451 441
Total cold run time: 55893 ms
Total hot run time: 52631 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4993 5081 4936 4936
q2 335 241 265 241
q3 4072 3956 4001 3956
q4 2762 2746 2729 2729
q5 6613 6492 6556 6492
q6 246 130 127 127
q7 3154 2726 2758 2726
q8 4815 4837 4811 4811
q9 17674 17712 17625 17625
q10 4070 4167 4132 4132
q11 743 650 622 622
q12 1011 824 827 824
q13 4315 3879 3928 3879
q14 390 365 339 339
q15 600 551 541 541
q16 753 727 706 706
q17 3915 3967 3960 3960
q18 9465 9296 9039 9039
q19 1852 1779 1784 1779
q20 2388 2049 2043 2043
q21 8823 8912 8808 8808
q22 948 896 855 855
Total cold run time: 83937 ms
Total hot run time: 81170 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1803917781
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 1546ee3708284566f690a63e1ab98476c4446876, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5231 5045 5120 5045
q2 369 185 198 185
q3 2109 2053 2022 2022
q4 1479 1424 1451 1424
q5 4174 4158 4136 4136
q6 260 134 130 130
q7 2077 1607 1634 1607
q8 2787 2772 2777 2772
q9 10469 10429 10293 10293
q10 3487 3550 3588 3550
q11 384 255 250 250
q12 463 312 301 301
q13 4550 4114 4127 4114
q14 334 306 284 284
q15 607 544 561 544
q16 700 626 589 589
q17 1157 1108 1095 1095
q18 7882 7369 7389 7369
q19 1726 1722 1718 1718
q20 596 360 371 360
q21 4984 4599 4602 4599
q22 538 439 428 428
Total cold run time: 56363 ms
Total hot run time: 52815 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4990 5011 5031 5011
q2 347 243 261 243
q3 3998 3977 4005 3977
q4 2789 2741 2734 2734
q5 6600 6512 6565 6512
q6 247 125 125 125
q7 3104 2744 2617 2617
q8 4823 4800 4802 4800
q9 17812 17575 17724 17575
q10 4099 4193 4170 4170
q11 745 623 636 623
q12 1004 824 835 824
q13 4294 3872 3913 3872
q14 381 352 360 352
q15 601 566 571 566
q16 771 695 697 695
q17 3970 3886 3933 3886
q18 9280 9347 9374 9347
q19 1791 1794 1805 1794
q20 2381 2070 2045 2045
q21 8803 8770 8596 8596
q22 964 846 835 835
Total cold run time: 83794 ms
Total hot run time: 81199 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805271786
TeamCity be ut coverage result:
Function Coverage: 36.76% (8405/22862)
Line Coverage: 29.29% (68157/232736)
Region Coverage: 27.93% (35235/126177)
Branch Coverage: 24.75% (18025/72840)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e51d9b2f5496ac59fa7430b05f69054b9ea2c4c1_e51d9b2f5496ac59fa7430b05f69054b9ea2c4c1/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange closed pull request #26711: [refactor](pipelineX) refine some read dependency
URL: https://github.com/apache/doris/pull/26711
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1803936422
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.3 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17162672244 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1804845569
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1804845171
run External
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1804849237
run external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805353333
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805452794
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805593576
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805392969
TeamCity be ut coverage result:
Function Coverage: 36.76% (8404/22862)
Line Coverage: 29.28% (68144/232736)
Region Coverage: 27.91% (35220/126177)
Branch Coverage: 24.74% (18022/72840)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0969fd6dad522aa855c6c0b5a5b592c438999759_0969fd6dad522aa855c6c0b5a5b592c438999759/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1803890335
TeamCity be ut coverage result:
Function Coverage: 36.73% (8394/22851)
Line Coverage: 29.24% (68084/232817)
Region Coverage: 27.90% (35220/126250)
Branch Coverage: 24.71% (18009/72896)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1546ee3708284566f690a63e1ab98476c4446876_1546ee3708284566f690a63e1ab98476c4446876/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26711:
URL: https://github.com/apache/doris/pull/26711#discussion_r1388015052
##########
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##########
@@ -87,10 +92,39 @@
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+ _set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
+ if (!_has_dependencys) {
+ return;
+ }
+ auto* dep = _dependencys[sender_idx];
+ DCHECK(dep);
+ dep->set_ready_for_read();
+}
+
+void MultiCastDataStreamer::_set_ready_for_read() {
Review Comment:
warning: method '_set_ready_for_read' can be made const [readability-make-member-function-const]
```suggestion
void MultiCastDataStreamer::_set_ready_for_read() const {
```
be/src/pipeline/exec/multi_cast_data_streamer.h:80:
```diff
- void _set_ready_for_read();
+ void _set_ready_for_read() const;
```
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -899,7 +885,30 @@
public:
ENABLE_FACTORY_CREATOR(LocalExchangeSharedState);
std::vector<moodycamel::ConcurrentQueue<PartitionedBlock>> data_queue;
+ std::vector<Dependency*> source_dependencies;
std::atomic<int> running_sink_operators = 0;
+ void add_running_sink_operators() { running_sink_operators++; }
+ void sub_running_sink_operators() {
+ running_sink_operators--;
+ if (running_sink_operators == 0) {
+ _set_ready_for_read();
+ }
+ }
+ void _set_ready_for_read() {
+ for (auto* dep : source_dependencies) {
+ DCHECK(dep);
+ dep->set_ready_for_read();
+ }
+ }
+ void set_dep_by_channel_id(Dependency* dep, int channel_id) {
+ source_dependencies[channel_id] = dep;
+ dep->block_reading();
+ }
+ void _set_ready_for_read(int channel_id) {
Review Comment:
warning: method '_set_ready_for_read' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void _set_ready_for_read(int channel_id) {
```
##########
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##########
@@ -87,10 +92,39 @@
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+ _set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
+ if (!_has_dependencys) {
+ return;
+ }
+ auto* dep = _dependencys[sender_idx];
+ DCHECK(dep);
+ dep->set_ready_for_read();
+}
+
+void MultiCastDataStreamer::_set_ready_for_read() {
+ if (!_has_dependencys) {
+ return;
+ }
+ for (auto* dep : _dependencys) {
+ DCHECK(dep);
+ dep->set_ready_for_read();
+ }
+}
+
+void MultiCastDataStreamer::_block_reading(int sender_idx) {
Review Comment:
warning: method '_block_reading' can be made const [readability-make-member-function-const]
```suggestion
void MultiCastDataStreamer::_block_reading(int sender_idx) const {
```
be/src/pipeline/exec/multi_cast_data_streamer.h:81:
```diff
- void _block_reading(int sender_idx);
+ void _block_reading(int sender_idx) const;
```
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -899,7 +885,30 @@ struct LocalExchangeSharedState {
public:
ENABLE_FACTORY_CREATOR(LocalExchangeSharedState);
std::vector<moodycamel::ConcurrentQueue<PartitionedBlock>> data_queue;
+ std::vector<Dependency*> source_dependencies;
std::atomic<int> running_sink_operators = 0;
+ void add_running_sink_operators() { running_sink_operators++; }
+ void sub_running_sink_operators() {
+ running_sink_operators--;
+ if (running_sink_operators == 0) {
+ _set_ready_for_read();
+ }
+ }
+ void _set_ready_for_read() {
+ for (auto* dep : source_dependencies) {
+ DCHECK(dep);
+ dep->set_ready_for_read();
+ }
+ }
+ void set_dep_by_channel_id(Dependency* dep, int channel_id) {
Review Comment:
warning: method 'set_dep_by_channel_id' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void set_dep_by_channel_id(Dependency* dep, int channel_id) {
```
##########
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##########
@@ -87,10 +92,39 @@ Status MultiCastDataStreamer::push(RuntimeState* state, doris::vectorized::Block
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+ _set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
Review Comment:
warning: method '_set_ready_for_read' can be made const [readability-make-member-function-const]
```suggestion
void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) const {
```
be/src/pipeline/exec/multi_cast_data_streamer.h:79:
```diff
- void _set_ready_for_read(int sender_idx);
+ void _set_ready_for_read(int sender_idx) const;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1803841506
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805317417
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit e51d9b2f5496ac59fa7430b05f69054b9ea2c4c1, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5286 5061 5166 5061
q2 367 180 200 180
q3 2094 2073 2079 2073
q4 1477 1447 1422 1422
q5 4163 4162 4113 4113
q6 255 136 135 135
q7 2081 1622 1618 1618
q8 2764 2765 2769 2765
q9 10634 10388 10296 10296
q10 3478 3561 3572 3561
q11 370 260 256 256
q12 449 318 304 304
q13 4539 4161 4152 4152
q14 325 294 292 292
q15 647 545 562 545
q16 698 615 602 602
q17 1144 1094 1098 1094
q18 7808 7358 7424 7358
q19 1707 1705 1715 1705
q20 593 373 346 346
q21 4949 4547 4582 4547
q22 553 434 443 434
Total cold run time: 56381 ms
Total hot run time: 52859 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4987 5007 5032 5007
q2 337 235 254 235
q3 4024 3970 3960 3960
q4 2792 2759 2752 2752
q5 6542 6470 6465 6465
q6 245 129 129 129
q7 3177 2727 2728 2727
q8 4816 4732 4746 4732
q9 17785 17744 17614 17614
q10 4073 4158 4186 4158
q11 751 654 638 638
q12 995 836 830 830
q13 4315 3947 3978 3947
q14 378 364 368 364
q15 598 559 567 559
q16 765 690 736 690
q17 3979 3870 3919 3870
q18 9327 9251 9055 9055
q19 1874 1776 1793 1776
q20 2409 2053 2055 2053
q21 8846 8697 8613 8613
q22 969 900 863 863
Total cold run time: 83984 ms
Total hot run time: 81037 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1805440701
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.16 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162360562 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipelineX) refine some read dependency [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #26711:
URL: https://github.com/apache/doris/pull/26711#issuecomment-1807017417
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org