You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2023/12/21 06:54:52 UTC
[PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Gabriel39 opened a new pull request, #28783:
URL: https://github.com/apache/doris/pull/28783
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #28783:
URL: https://github.com/apache/doris/pull/28783#issuecomment-1865672897
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28783:
URL: https://github.com/apache/doris/pull/28783#issuecomment-1865845262
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 8ce113ce81549a749b73df3d15458edb82b2849a, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4661 4419 4429 4419
q2 367 148 158 148
q3 1466 1254 1198 1198
q4 1108 936 919 919
q5 3153 3166 3169 3166
q6 246 128 130 128
q7 998 482 482 482
q8 2189 2228 2181 2181
q9 6669 6662 6645 6645
q10 3211 3308 3319 3308
q11 302 190 184 184
q12 353 218 208 208
q13 4586 3782 3790 3782
q14 243 213 215 213
q15 569 519 528 519
q16 436 386 393 386
q17 997 571 575 571
q18 7217 6883 6890 6883
q19 1518 1353 1401 1353
q20 505 311 300 300
q21 3062 2628 2672 2628
q22 351 283 292 283
Total cold run time: 44207 ms
Total hot run time: 39904 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4367 4329 4324 4324
q2 272 163 181 163
q3 3542 3532 3526 3526
q4 2390 2377 2388 2377
q5 5738 5727 5717 5717
q6 242 125 123 123
q7 2388 1874 1853 1853
q8 3516 3530 3522 3522
q9 9034 9004 8982 8982
q10 3922 4007 4001 4001
q11 495 381 378 378
q12 767 588 600 588
q13 4300 3562 3557 3557
q14 292 259 247 247
q15 577 517 518 517
q16 526 462 482 462
q17 1873 1861 1869 1861
q18 8680 8185 8273 8185
q19 1700 1732 1728 1728
q20 2254 1938 1934 1934
q21 6537 6188 6220 6188
q22 517 430 418 418
Total cold run time: 63929 ms
Total hot run time: 60651 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28783:
URL: https://github.com/apache/doris/pull/28783#issuecomment-1865903214
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28783:
URL: https://github.com/apache/doris/pull/28783#issuecomment-1865810913
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.08 seconds
stream load tsv: 569 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17184091514 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28783:
URL: https://github.com/apache/doris/pull/28783#issuecomment-1865903163
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #28783:
URL: https://github.com/apache/doris/pull/28783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28783:
URL: https://github.com/apache/doris/pull/28783#issuecomment-1865773321
TeamCity be ut coverage result:
Function Coverage: 36.48% (8537/23405)
Line Coverage: 28.60% (69427/242783)
Region Coverage: 27.62% (35929/130081)
Branch Coverage: 24.36% (18366/75384)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8ce113ce81549a749b73df3d15458edb82b2849a_8ce113ce81549a749b73df3d15458edb82b2849a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](fix) Fix TPCH Q2 [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28783:
URL: https://github.com/apache/doris/pull/28783#discussion_r1433627644
##########
be/src/pipeline/pipeline.h:
##########
@@ -122,14 +122,30 @@ class Pipeline : public std::enable_shared_from_this<Pipeline> {
return _collect_query_statistics_with_every_batch;
}
- bool need_to_local_shuffle(const DataDistribution target_data_distribution) const {
- if (target_data_distribution.distribution_type == ExchangeType::BUCKET_HASH_SHUFFLE ||
- target_data_distribution.distribution_type == ExchangeType::HASH_SHUFFLE) {
- // If `_data_distribution` of this pipeline does not match the `target_data_distribution`,
- // we should do local shuffle.
- return target_data_distribution.operator!=(_data_distribution);
+ static bool is_hash_exchange(ExchangeType idx) {
+ return idx == ExchangeType::HASH_SHUFFLE || idx == ExchangeType::BUCKET_HASH_SHUFFLE;
+ }
+
+ bool need_to_local_exchange(const DataDistribution target_data_distribution) const {
Review Comment:
warning: method 'need_to_local_exchange' can be made static [readability-convert-member-functions-to-static]
```suggestion
static bool need_to_local_exchange(const DataDistribution target_data_distribution) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org