You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2023/12/01 02:40:11 UTC
[PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Gabriel39 opened a new pull request, #27852:
URL: https://github.com/apache/doris/pull/27852
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27852:
URL: https://github.com/apache/doris/pull/27852#issuecomment-1837134239
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27852:
URL: https://github.com/apache/doris/pull/27852#discussion_r1411829012
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h:
##########
@@ -60,10 +60,9 @@ class LocalExchangeSourceLocalState final
class LocalExchangeSourceOperatorX final : public OperatorX<LocalExchangeSourceLocalState> {
public:
using Base = OperatorX<LocalExchangeSourceLocalState>;
- LocalExchangeSourceOperatorX(ObjectPool* pool, int id, OperatorXBase* parent)
- : Base(pool, -1, id), _parent(parent) {}
- Status init(const TPlanNode& tnode, RuntimeState* state) override {
- _op_name = "LOCAL_EXCHANGE_OPERATOR";
+ LocalExchangeSourceOperatorX(ObjectPool* pool, int id) : Base(pool, -1, id) {}
+ Status init(ExchangeType type) override {
Review Comment:
warning: method 'init' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status init(ExchangeType type) override {
```
##########
be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:
##########
@@ -606,49 +634,100 @@ Status PipelineXFragmentContext::_create_tree_helper(ObjectPool* pool,
return Status::OK();
}
-Status PipelineXFragmentContext::_add_local_exchange(ObjectPool* pool, OperatorXPtr& op,
- PipelinePtr& cur_pipe, const TPlanNode& tnode,
+Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
Review Comment:
warning: function '_add_local_exchange' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:636:** 91 lines including whitespace and comments (threshold 80)
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27852:
URL: https://github.com/apache/doris/pull/27852#discussion_r1411739608
##########
be/src/pipeline/exec/exchange_sink_operator.cpp:
##########
@@ -245,7 +245,7 @@ Status ExchangeSinkLocalState::open(RuntimeState* state) {
return Status::OK();
}
-std::string ExchangeSinkLocalState::id_name() {
+std::string ExchangeSinkLocalState::name_suffix() {
Review Comment:
warning: method 'name_suffix' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/exchange_sink_operator.h:186:
```diff
- std::string name_suffix() override;
+ static std::string name_suffix() override;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27852:
URL: https://github.com/apache/doris/pull/27852#issuecomment-1836100974
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 8ed991752d6a87ce368d68b5a52f6ee2968b2256, data reload: true
run tpch-sf100 query with default conf and session variables
q1 4844 4590 4612 4590
q2 377 123 121 121
q3 1531 1309 1260 1260
q4 1160 953 951 951
q5 3294 3267 3237 3237
q6 243 123 122 122
q7 980 505 505 505
q8 2247 2223 2214 2214
q9 6992 6932 6945 6932
q10 3327 3393 3370 3370
q11 341 205 206 205
q12 363 217 210 210
q13 4687 3878 3851 3851
q14 240 202 207 202
q15 592 534 522 522
q16 455 411 375 375
q17 1008 622 635 622
q18 7910 7371 7493 7371
q19 1574 1553 1573 1553
q20 513 363 321 321
q21 3447 2957 2985 2957
q22 364 293 291 291
Total cold run time: 46489 ms
Total hot run time: 41782 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4603 4595 4599 4595
q2 318 239 236 236
q3 3742 3758 3749 3749
q4 2525 2513 2517 2513
q5 6201 6251 6199 6199
q6 244 123 122 122
q7 2607 2041 1974 1974
q8 3766 3732 3680 3680
q9 9512 9405 9406 9405
q10 4063 4173 4182 4173
q11 646 510 508 508
q12 811 673 622 622
q13 4374 3616 3612 3612
q14 286 246 262 246
q15 586 524 527 524
q16 507 494 473 473
q17 2151 2119 2093 2093
q18 9539 8923 9030 8923
q19 1783 1758 1776 1758
q20 2312 2003 1968 1968
q21 7211 6925 6978 6925
q22 670 551 554 551
Total cold run time: 68457 ms
Total hot run time: 64849 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #27852:
URL: https://github.com/apache/doris/pull/27852#issuecomment-1835750219
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27852:
URL: https://github.com/apache/doris/pull/27852#discussion_r1411764557
##########
be/src/pipeline/exec/hashjoin_build_sink.h:
##########
@@ -155,6 +155,14 @@ class HashJoinBuildSinkOperatorX final
._should_build_hash_table;
}
+ std::vector<TExpr> get_local_shuffle_exprs() const override { return _partition_exprs; }
+ ExchangeType get_local_exchange_type() const override {
Review Comment:
warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]
```suggestion
static ExchangeType get_local_exchange_type() override {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #27852:
URL: https://github.com/apache/doris/pull/27852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27852:
URL: https://github.com/apache/doris/pull/27852#discussion_r1411611309
##########
be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:
##########
@@ -606,49 +633,102 @@ Status PipelineXFragmentContext::_create_tree_helper(ObjectPool* pool,
return Status::OK();
}
-Status PipelineXFragmentContext::_add_local_exchange(ObjectPool* pool, OperatorXPtr& op,
- PipelinePtr& cur_pipe, const TPlanNode& tnode,
+Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
Review Comment:
warning: function '_add_local_exchange' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:635:** 93 lines including whitespace and comments (threshold 80)
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27852:
URL: https://github.com/apache/doris/pull/27852#discussion_r1411731648
##########
be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.h:
##########
@@ -110,6 +110,7 @@ class DistinctStreamingAggSinkOperatorX final
Status init(const TPlanNode& tnode, RuntimeState* state) override;
Status sink(RuntimeState* state, vectorized::Block* in_block,
SourceState source_state) override;
+ ExchangeType get_local_exchange_type() const override { return ExchangeType::PASSTHROUGH; }
Review Comment:
warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]
```suggestion
static ExchangeType get_local_exchange_type() override { return ExchangeType::PASSTHROUGH; }
```
##########
be/src/pipeline/exec/streaming_aggregation_sink_operator.h:
##########
@@ -120,6 +120,7 @@ class StreamingAggSinkOperatorX final : public AggSinkOperatorX<StreamingAggSink
Status init(const TPlanNode& tnode, RuntimeState* state) override;
Status sink(RuntimeState* state, vectorized::Block* in_block,
SourceState source_state) override;
+ ExchangeType get_local_exchange_type() const override { return ExchangeType::PASSTHROUGH; }
Review Comment:
warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]
```suggestion
static ExchangeType get_local_exchange_type() override { return ExchangeType::PASSTHROUGH; }
```
##########
be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:
##########
@@ -606,49 +634,102 @@ Status PipelineXFragmentContext::_create_tree_helper(ObjectPool* pool,
return Status::OK();
}
-Status PipelineXFragmentContext::_add_local_exchange(ObjectPool* pool, OperatorXPtr& op,
- PipelinePtr& cur_pipe, const TPlanNode& tnode,
+Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
Review Comment:
warning: function '_add_local_exchange' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:636:** 93 lines including whitespace and comments (threshold 80)
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #27852:
URL: https://github.com/apache/doris/pull/27852#issuecomment-1837134251
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #27852:
URL: https://github.com/apache/doris/pull/27852#issuecomment-1835921340
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.85 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17163861985 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](improvement) Support local shuffle for join and agg [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #27852:
URL: https://github.com/apache/doris/pull/27852#discussion_r1411539334
##########
be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:
##########
@@ -606,49 +623,100 @@ Status PipelineXFragmentContext::_create_tree_helper(ObjectPool* pool,
return Status::OK();
}
-Status PipelineXFragmentContext::_add_local_exchange(ObjectPool* pool, OperatorXPtr& op,
- PipelinePtr& cur_pipe, const TPlanNode& tnode,
+Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
Review Comment:
warning: function '_add_local_exchange' exceeds recommended size/complexity thresholds [readability-function-size]
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
<details>
<summary>Additional context</summary>
**be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp:625:** 92 lines including whitespace and comments (threshold 80)
```cpp
Status PipelineXFragmentContext::_add_local_exchange(int idx, int node_id, ObjectPool* pool,
^
```
</details>
##########
be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h:
##########
@@ -60,9 +60,8 @@ class LocalExchangeSourceLocalState final
class LocalExchangeSourceOperatorX final : public OperatorX<LocalExchangeSourceLocalState> {
public:
using Base = OperatorX<LocalExchangeSourceLocalState>;
- LocalExchangeSourceOperatorX(ObjectPool* pool, int id, OperatorXBase* parent)
- : Base(pool, -1, id), _parent(parent) {}
- Status init(const TPlanNode& tnode, RuntimeState* state) override {
+ LocalExchangeSourceOperatorX(ObjectPool* pool, int id) : Base(pool, -1, id) {}
+ Status init() override {
Review Comment:
warning: method 'init' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Status init() override {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org