You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/12/11 09:30:45 UTC
[PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Mryange opened a new pull request, #28248:
URL: https://github.com/apache/doris/pull/28248
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849865890
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1851066294
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1851095239
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit a51ef03bb7ac5b147141fe3cba2b91cd07ac4783, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4733 4480 4708 4480
q2 363 153 162 153
q3 1446 1203 1195 1195
q4 1112 952 874 874
q5 3120 3120 3139 3120
q6 250 127 127 127
q7 979 495 479 479
q8 2226 2208 2190 2190
q9 6703 6696 6669 6669
q10 3221 3277 3300 3277
q11 325 206 198 198
q12 343 211 211 211
q13 4563 3863 3814 3814
q14 242 208 206 206
q15 563 521 527 521
q16 455 383 389 383
q17 1011 585 534 534
q18 7546 7094 7103 7094
q19 1521 1311 1463 1311
q20 533 300 320 300
q21 3038 2660 2670 2660
q22 351 277 279 277
Total cold run time: 44644 ms
Total hot run time: 40073 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4413 4400 4381 4381
q2 267 164 171 164
q3 3552 3557 3537 3537
q4 2392 2377 2380 2377
q5 5745 5735 5760 5735
q6 243 121 122 121
q7 2416 1872 1912 1872
q8 3536 3519 3530 3519
q9 9052 9000 8971 8971
q10 3885 4009 4006 4006
q11 494 393 373 373
q12 763 598 594 594
q13 4314 3573 3554 3554
q14 289 253 252 252
q15 569 518 517 517
q16 516 468 476 468
q17 1891 1860 1862 1860
q18 8808 8332 8373 8332
q19 1794 1763 1741 1741
q20 2265 1946 1935 1935
q21 6538 6199 6155 6155
q22 496 413 437 413
Total cold run time: 64238 ms
Total hot run time: 60877 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849648159
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849678386
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1850151504
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit a51ef03bb7ac5b147141fe3cba2b91cd07ac4783, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4711 4506 4457 4457
q2 363 147 157 147
q3 1456 1270 1211 1211
q4 1098 936 843 843
q5 3139 3162 3173 3162
q6 254 127 128 127
q7 1009 491 474 474
q8 2207 2229 2188 2188
q9 6650 6686 6647 6647
q10 3241 3260 3255 3255
q11 331 194 194 194
q12 347 200 205 200
q13 4632 4258 3797 3797
q14 241 215 209 209
q15 574 519 521 519
q16 444 385 384 384
q17 1001 554 541 541
q18 7601 7531 7827 7531
q19 1517 1333 1452 1333
q20 546 310 285 285
q21 3074 2648 2738 2648
q22 342 273 286 273
Total cold run time: 44778 ms
Total hot run time: 40425 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4402 4409 4394 4394
q2 266 160 173 160
q3 3553 3547 3538 3538
q4 2388 2376 2375 2375
q5 5753 5760 5752 5752
q6 241 121 120 120
q7 2372 1904 1877 1877
q8 3533 3536 3527 3527
q9 9030 9012 8944 8944
q10 3890 4008 3991 3991
q11 494 380 376 376
q12 763 588 618 588
q13 4312 3552 3568 3552
q14 283 258 259 258
q15 566 525 513 513
q16 502 450 472 450
q17 1884 1840 1856 1840
q18 8763 8273 8301 8273
q19 1731 1735 1723 1723
q20 2263 1942 1920 1920
q21 6561 6213 6222 6213
q22 513 412 415 412
Total cold run time: 64063 ms
Total hot run time: 60796 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849933787
run pipelinex_p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849650153
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849755931
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit a51ef03bb7ac5b147141fe3cba2b91cd07ac4783, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4598 4450 4481 4450
q2 362 148 162 148
q3 1450 1205 1221 1205
q4 1113 942 904 904
q5 3119 3127 3132 3127
q6 250 128 125 125
q7 1006 486 471 471
q8 2218 2236 2183 2183
q9 6701 7205 6661 6661
q10 3199 3258 3253 3253
q11 320 208 207 207
q12 352 213 211 211
q13 4640 4011 3824 3824
q14 239 216 222 216
q15 563 525 523 523
q16 440 392 397 392
q17 1024 594 564 564
q18 8351 7585 7888 7585
q19 1522 1375 1460 1375
q20 541 315 287 287
q21 3085 2655 2686 2655
q22 349 282 283 282
Total cold run time: 45442 ms
Total hot run time: 40648 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4404 4381 4415 4381
q2 266 164 166 164
q3 3551 3559 3529 3529
q4 2377 2383 2379 2379
q5 5743 5744 5740 5740
q6 241 121 121 121
q7 2429 1909 1879 1879
q8 3524 3526 3542 3526
q9 9079 8985 8999 8985
q10 3918 3971 3997 3971
q11 506 391 379 379
q12 771 586 587 586
q13 4283 3569 3537 3537
q14 291 254 259 254
q15 576 509 518 509
q16 503 439 464 439
q17 1861 1843 1847 1843
q18 8790 8528 8348 8348
q19 1786 1807 1786 1786
q20 2284 1943 1935 1935
q21 6572 6194 6171 6171
q22 498 419 416 416
Total cold run time: 64253 ms
Total hot run time: 60878 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28248:
URL: https://github.com/apache/doris/pull/28248#discussion_r1422190889
##########
be/src/pipeline/exec/partition_sort_sink_operator.h:
##########
@@ -105,6 +105,12 @@ class PartitionSortSinkOperatorX final : public DataSinkOperatorX<PartitionSortS
Status open(RuntimeState* state) override;
Status sink(RuntimeState* state, vectorized::Block* in_block,
SourceState source_state) override;
+ ExchangeType get_local_exchange_type() const override {
Review Comment:
warning: method 'get_local_exchange_type' can be made static [readability-convert-member-functions-to-static]
```suggestion
static ExchangeType get_local_exchange_type() override {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange closed pull request #28248: [only test](pipelineX) add local_shuffle in partition sort node
URL: https://github.com/apache/doris/pull/28248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1850101884
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1849768349
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.24 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17222925735 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1850164479
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.86 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17219793564 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [only test](pipelineX) add local_shuffle in partition sort node [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28248:
URL: https://github.com/apache/doris/pull/28248#issuecomment-1851105870
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.64 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17218035322 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org