You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/12/12 03:10:47 UTC
[PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Mryange opened a new pull request, #28265:
URL: https://github.com/apache/doris/pull/28265
## Proposed changes
test in
https://github.com/apache/doris/pull/28254
https://github.com/apache/doris/pull/28248
https://github.com/apache/doris/pull/28146
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28265:
URL: https://github.com/apache/doris/pull/28265#issuecomment-1851290037
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.08 seconds
stream load tsv: 597 seconds loaded 74807831229 Bytes, about 119 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17224361124 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28265:
URL: https://github.com/apache/doris/pull/28265#issuecomment-1851232701
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28265:
URL: https://github.com/apache/doris/pull/28265#issuecomment-1851315962
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28265:
URL: https://github.com/apache/doris/pull/28265#issuecomment-1851295010
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit c28f48966a0940ded8f9cfc060289c5319e218c5, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4648 4446 4446 4446
q2 369 146 162 146
q3 1456 1229 1184 1184
q4 1117 899 883 883
q5 3117 3180 3130 3130
q6 249 124 128 124
q7 1015 484 475 475
q8 2212 2233 2208 2208
q9 6656 6676 6679 6676
q10 3190 3256 3261 3256
q11 328 208 209 208
q12 347 201 206 201
q13 4625 3854 3834 3834
q14 243 210 210 210
q15 562 526 521 521
q16 436 388 401 388
q17 1018 555 514 514
q18 8999 7889 8763 7889
q19 1520 1269 1396 1269
q20 507 330 319 319
q21 3069 2674 2697 2674
q22 352 274 284 274
Total cold run time: 46035 ms
Total hot run time: 40829 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4362 4392 4390 4390
q2 269 164 172 164
q3 3555 3536 3538 3536
q4 2397 2376 2383 2376
q5 5758 5763 5766 5763
q6 238 120 122 120
q7 2394 1874 1854 1854
q8 3538 3533 3537 3533
q9 9096 9036 9007 9007
q10 3914 3982 3989 3982
q11 502 391 414 391
q12 764 597 614 597
q13 4308 3541 3552 3541
q14 284 254 248 248
q15 574 510 518 510
q16 509 452 467 452
q17 1884 1835 1851 1835
q18 8791 8354 8391 8354
q19 1756 1781 1758 1758
q20 2251 1969 1931 1931
q21 6550 6203 6154 6154
q22 493 429 421 421
Total cold run time: 64187 ms
Total hot run time: 60917 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28265:
URL: https://github.com/apache/doris/pull/28265#issuecomment-1851623641
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28265:
URL: https://github.com/apache/doris/pull/28265#issuecomment-1851623751
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28265:
URL: https://github.com/apache/doris/pull/28265#discussion_r1423357240
##########
be/src/pipeline/exec/partition_sort_sink_operator.h:
##########
@@ -105,6 +105,12 @@ class PartitionSortSinkOperatorX final : public DataSinkOperatorX<PartitionSortS
Status open(RuntimeState* state) override;
Status sink(RuntimeState* state, vectorized::Block* in_block,
SourceState source_state) override;
+ ExchangeType get_local_exchange_type() const override {
Review Comment:
warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]
```suggestion
static ExchangeType get_local_exchange_type() override {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [feature](pipelineX) add local_shuffle in sort partition sort analytic node [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #28265:
URL: https://github.com/apache/doris/pull/28265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org