You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2023/11/13 07:49:45 UTC
[PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Gabriel39 opened a new pull request, #26879:
URL: https://github.com/apache/doris/pull/26879
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1807879136
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814736564
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.96 seconds
stream load tsv: 576 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17101304669 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812781004
TeamCity be ut coverage result:
Function Coverage: 36.76% (8414/22888)
Line Coverage: 29.26% (68395/233735)
Region Coverage: 27.89% (35360/126762)
Branch Coverage: 24.66% (18072/73278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bc604c10a3eb05d367f99a7ad7875104b7978962_bc604c10a3eb05d367f99a7ad7875104b7978962/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815724700
TeamCity be ut coverage result:
Function Coverage: 36.71% (8412/22916)
Line Coverage: 29.23% (68401/234030)
Region Coverage: 27.84% (35352/126993)
Branch Coverage: 24.61% (18063/73398)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fcd20a9e735b5bddb289cba38b7458d95f6bc6bf_fcd20a9e735b5bddb289cba38b7458d95f6bc6bf/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816026305
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816192773
TeamCity be ut coverage result:
Function Coverage: 36.71% (8414/22922)
Line Coverage: 29.24% (68442/234079)
Region Coverage: 27.84% (35376/127067)
Branch Coverage: 24.60% (18067/73442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/dba316194c913b6861a9adc40d809962d05a7cac_dba316194c913b6861a9adc40d809962d05a7cac/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816660544
TeamCity be ut coverage result:
Function Coverage: 36.74% (8434/22953)
Line Coverage: 29.26% (68608/234485)
Region Coverage: 27.87% (35472/127273)
Branch Coverage: 24.60% (18097/73554)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0a03b082af048287e5ad39f90a8404c8adbc3725_0a03b082af048287e5ad39f90a8404c8adbc3725/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #26879:
URL: https://github.com/apache/doris/pull/26879
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1807992439
TeamCity be ut coverage result:
Function Coverage: 36.74% (8405/22880)
Line Coverage: 29.27% (68140/232776)
Region Coverage: 27.86% (35171/126261)
Branch Coverage: 24.69% (17980/72830)
Coverage Report: http://coverage.selectdb-in.cc/coverage/040362189d4e122606294adb6eee92f23ecbb4c5_040362189d4e122606294adb6eee92f23ecbb4c5/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1807916847
TeamCity be ut coverage result:
Function Coverage: 36.73% (8405/22884)
Line Coverage: 29.27% (68131/232770)
Region Coverage: 27.86% (35171/126262)
Branch Coverage: 24.68% (17973/72828)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b3f5836ff6f7fc513beda86be99bb0536ea4596d_b3f5836ff6f7fc513beda86be99bb0536ea4596d/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813824401
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814398335
TeamCity be ut coverage result:
Function Coverage: 36.72% (8411/22904)
Line Coverage: 29.25% (68409/233889)
Region Coverage: 27.86% (35354/126915)
Branch Coverage: 24.63% (18067/73360)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8640ca36b30155ec174e9f69258bac80a91dfdf1_8640ca36b30155ec174e9f69258bac80a91dfdf1/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809878271
TeamCity be ut coverage result:
Function Coverage: 36.72% (8419/22926)
Line Coverage: 29.25% (68405/233825)
Region Coverage: 27.87% (35349/126841)
Branch Coverage: 24.67% (18073/73252)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bc47d83080de1b2b497e32c426fc6d22a480e5a6_bc47d83080de1b2b497e32c426fc6d22a480e5a6/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812538091
TeamCity be ut coverage result:
Function Coverage: 36.76% (8416/22894)
Line Coverage: 29.24% (68401/233893)
Region Coverage: 27.87% (35356/126876)
Branch Coverage: 24.65% (18070/73310)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ac5c20d765a2ad705701454cefe503062ed93c70_ac5c20d765a2ad705701454cefe503062ed93c70/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812856618
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.47 seconds
stream load tsv: 575 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098816500 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812931739
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816878670
TeamCity be ut coverage result:
Function Coverage: 36.75% (8434/22947)
Line Coverage: 29.26% (68614/234470)
Region Coverage: 27.87% (35476/127271)
Branch Coverage: 24.61% (18100/73558)
Coverage Report: http://coverage.selectdb-in.cc/coverage/37e4bef18594c926b2b21a18b183c3eb26a70713_37e4bef18594c926b2b21a18b183c3eb26a70713/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815683338
TeamCity be ut coverage result:
Function Coverage: 36.72% (8415/22916)
Line Coverage: 29.25% (68448/234022)
Region Coverage: 27.86% (35374/126993)
Branch Coverage: 24.62% (18070/73398)
Coverage Report: http://coverage.selectdb-in.cc/coverage/de10a13680fe93b851d8abc215fc90ae0782110a_de10a13680fe93b851d8abc215fc90ae0782110a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812976557
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 3728ada251f28b1d873e32419b8b339d5ceaeaa3, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4905 4643 4673 4643
q2 360 180 163 163
q3 2076 1948 1906 1906
q4 1394 1289 1247 1247
q5 3941 3931 4031 3931
q6 255 129 131 129
q7 1405 886 885 885
q8 2767 2782 2778 2778
q9 9880 9630 9541 9541
q10 3463 3531 3500 3500
q11 384 249 251 249
q12 445 290 294 290
q13 4609 3807 3834 3807
q14 322 277 284 277
q15 585 534 537 534
q16 666 584 578 578
q17 1126 983 925 925
q18 7839 7384 7408 7384
q19 1683 1650 1667 1650
q20 567 320 311 311
q21 4397 3974 3978 3974
q22 474 370 386 370
Total cold run time: 53543 ms
Total hot run time: 49072 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4606 4572 4592 4572
q2 352 248 262 248
q3 4040 3983 3981 3981
q4 2704 2682 2685 2682
q5 9506 9571 9496 9496
q6 245 124 125 124
q7 2619 2296 2277 2277
q8 4396 4455 4457 4455
q9 13163 13117 13128 13117
q10 4101 4201 4186 4186
q11 779 646 641 641
q12 969 826 811 811
q13 4321 3548 3613 3548
q14 381 356 349 349
q15 585 522 535 522
q16 732 663 670 663
q17 3942 3914 3889 3889
q18 9562 9052 8986 8986
q19 1800 1774 1763 1763
q20 2376 2059 2037 2037
q21 8770 8529 8846 8529
q22 859 815 829 815
Total cold run time: 80808 ms
Total hot run time: 77691 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816759182
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816906795
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814471473
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1395742372
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +449,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:783:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1392440089
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +427,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:801:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809578120
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 63a5e6c475055ca313dde96539ee9554ac0bcc55, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5270 5193 5300 5193
q2 366 156 160 156
q3 2072 2043 2007 2007
q4 1392 1373 1361 1361
q5 3985 3997 4005 3997
q6 252 132 132 132
q7 1491 890 893 890
q8 2780 2792 2770 2770
q9 9762 9463 9408 9408
q10 3474 3532 3549 3532
q11 391 256 240 240
q12 428 285 277 277
q13 4613 4159 4163 4159
q14 313 294 285 285
q15 618 573 549 549
q16 697 585 591 585
q17 1129 1075 1103 1075
q18 8007 7630 7612 7612
q19 1678 1709 1685 1685
q20 591 335 290 290
q21 4692 4395 4422 4395
q22 507 421 414 414
Total cold run time: 54508 ms
Total hot run time: 51012 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5011 5036 5097 5036
q2 332 226 249 226
q3 4094 4017 4022 4017
q4 2804 2781 2770 2770
q5 9715 9721 9739 9721
q6 244 120 123 120
q7 3041 2499 2450 2450
q8 4896 4850 4817 4817
q9 13148 13022 12972 12972
q10 4055 4165 4178 4165
q11 786 653 659 653
q12 981 862 843 843
q13 4292 3879 3852 3852
q14 385 335 355 335
q15 607 565 543 543
q16 749 670 685 670
q17 3925 3904 3936 3904
q18 9697 9432 9418 9418
q19 1845 1795 1772 1772
q20 2406 2065 2061 2061
q21 8906 8814 8901 8814
q22 972 835 861 835
Total cold run time: 82891 ms
Total hot run time: 79994 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812402603
TeamCity be ut coverage result:
Function Coverage: 36.77% (8417/22893)
Line Coverage: 29.25% (68402/233874)
Region Coverage: 27.87% (35356/126862)
Branch Coverage: 24.65% (18072/73306)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c9ca35f9d950007ca5fd6b94e320a87307cf398a_c9ca35f9d950007ca5fd6b94e320a87307cf398a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813802245
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813872204
TeamCity be ut coverage result:
Function Coverage: 36.76% (8413/22889)
Line Coverage: 29.27% (68441/233787)
Region Coverage: 27.88% (35367/126839)
Branch Coverage: 24.65% (18075/73326)
Coverage Report: http://coverage.selectdb-in.cc/coverage/957686ebe4d4cdeee40c584198102088c823ad47_957686ebe4d4cdeee40c584198102088c823ad47/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814459271
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.35 seconds
stream load tsv: 566 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098590874 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814474931
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815649596
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816640840
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814359755
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816057024
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812857424
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813772985
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.84 seconds
stream load tsv: 584 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17099230996 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1807987339
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 040362189d4e122606294adb6eee92f23ecbb4c5, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5299 5150 5129 5129
q2 378 184 164 164
q3 2064 2021 2005 2005
q4 1408 1361 1375 1361
q5 4002 3992 3989 3989
q6 254 130 131 130
q7 1500 904 911 904
q8 2786 2778 2772 2772
q9 9541 9669 9432 9432
q10 3454 3542 3548 3542
q11 377 254 259 254
q12 435 275 291 275
q13 4617 4138 4154 4138
q14 320 287 297 287
q15 631 574 577 574
q16 682 599 582 582
q17 1134 1083 1095 1083
q18 8010 7646 7603 7603
q19 1687 1693 1692 1692
q20 568 323 318 318
q21 4726 4398 4392 4392
q22 512 409 416 409
Total cold run time: 54385 ms
Total hot run time: 51035 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5020 5049 5214 5049
q2 336 224 237 224
q3 4149 3970 4052 3970
q4 2807 2808 2780 2780
q5 9756 9621 9675 9621
q6 245 127 128 127
q7 3092 2491 2527 2491
q8 4917 4885 4837 4837
q9 13064 13008 12988 12988
q10 4096 4212 4208 4208
q11 766 644 671 644
q12 982 859 862 859
q13 4278 3919 3904 3904
q14 384 336 361 336
q15 641 570 538 538
q16 791 720 667 667
q17 3950 3940 3874 3874
q18 9692 9546 9513 9513
q19 1881 1805 1768 1768
q20 2380 2075 2059 2059
q21 9081 8917 8775 8775
q22 906 851 880 851
Total cold run time: 83214 ms
Total hot run time: 80083 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812215062
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1394142218
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +451,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
+ if (child_id + 1 < probe_finished_children_dependency.size()) {
+ probe_finished_children_dependency[child_id + 1]->set_ready_for_write();
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:785:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -433,9 +596,19 @@
}
void RuntimeFilterDependency::sub_filters() {
- _filters--;
- if (_filters == 0) {
- *_blocked_by_rf = false;
+ auto value = _filters.fetch_sub(1);
+ if (value == 1) {
Review Comment:
warning: method 'sub_filters' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:258:
```diff
- void sub_filters();
+ static void sub_filters();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812591916
TeamCity be ut coverage result:
Function Coverage: 36.75% (8411/22888)
Line Coverage: 29.25% (68392/233828)
Region Coverage: 27.88% (35344/126787)
Branch Coverage: 24.66% (18071/73276)
Coverage Report: http://coverage.selectdb-in.cc/coverage/33cbfbecc15c5bdebeb7117ef79c947e7d559ae7_33cbfbecc15c5bdebeb7117ef79c947e7d559ae7/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812935037
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 203f07e7d7ecc417a03ba75c35f550a37c16e9a2, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4922 4704 4648 4648
q2 362 152 158 152
q3 2043 1992 1925 1925
q4 1400 1260 1239 1239
q5 3951 3908 4022 3908
q6 250 137 136 136
q7 1409 885 874 874
q8 2759 2776 2752 2752
q9 9994 9655 9599 9599
q10 3478 3541 3499 3499
q11 381 254 253 253
q12 435 297 296 296
q13 4599 3779 3764 3764
q14 311 285 297 285
q15 585 532 525 525
q16 676 567 575 567
q17 1117 988 905 905
q18 7750 7370 7488 7370
q19 1660 1674 1664 1664
q20 528 304 294 294
q21 4403 4038 3971 3971
q22 470 378 380 378
Total cold run time: 53483 ms
Total hot run time: 49004 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4574 4587 4610 4587
q2 344 221 250 221
q3 4016 3984 3996 3984
q4 2704 2691 2693 2691
q5 9555 9559 9650 9559
q6 249 126 132 126
q7 2603 2285 2283 2283
q8 4444 4448 4425 4425
q9 13160 13083 13119 13083
q10 4072 4176 4177 4176
q11 820 644 631 631
q12 986 811 835 811
q13 4308 3564 3535 3535
q14 371 343 362 343
q15 571 532 525 525
q16 721 676 661 661
q17 3857 3863 4015 3863
q18 9549 9025 9044 9025
q19 1827 1788 1778 1778
q20 2401 2066 2044 2044
q21 8846 8660 8626 8626
q22 875 786 791 786
Total cold run time: 80853 ms
Total hot run time: 77763 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1392382189
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +358,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:784:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,52 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:101:
```diff
- void add_block_task(PipelineXTask* task);
+ static void add_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,52 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ DCHECK(avoid_using_blocked_queue(task->get_state()));
+ DCHECK(_blocked_task.empty() || _blocked_task[_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK((task->get_state() == PipelineTaskState::BLOCKED_FOR_SINK && is_write_dependency() &&
+ ((WriteDependency*)this)->write_blocked_by() != nullptr) ||
+ (task->get_state() == PipelineTaskState::BLOCKED_FOR_RF && is_filter_dependency() &&
+ ((RuntimeFilterDependency*)this)->filter_blocked_by() != nullptr) ||
+ (task->get_state() == PipelineTaskState::PENDING_FINISH && is_finish_dependency() &&
+ ((FinishDependency*)this)->finish_blocked_by() != nullptr))
+ << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ _blocked_task.push_back(task);
+}
+
+void Dependency::_try_to_wake_up_task() {
Review Comment:
warning: method '_try_to_wake_up_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:104:
```diff
- void _try_to_wake_up_task();
+ static void _try_to_wake_up_task();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809970606
TeamCity be ut coverage result:
Function Coverage: 36.73% (8420/22926)
Line Coverage: 29.26% (68418/233823)
Region Coverage: 27.89% (35371/126839)
Branch Coverage: 24.67% (18072/73250)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ec2ad75622ad081d27207d40f4b428c33b4948f0_ec2ad75622ad081d27207d40f4b428c33b4948f0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812637612
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813765334
TeamCity be ut coverage result:
Function Coverage: 36.76% (8414/22889)
Line Coverage: 29.27% (68399/233716)
Region Coverage: 27.89% (35360/126773)
Branch Coverage: 24.66% (18070/73290)
Coverage Report: http://coverage.selectdb-in.cc/coverage/73f72606ea427a8c86820ae8995b47e8e5a0235e_73f72606ea427a8c86820ae8995b47e8e5a0235e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809752965
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816825236
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.42 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17101330979 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815782873
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815855816
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816587167
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816847782
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1397524667
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +500,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:737:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816681796
TeamCity be ut coverage result:
Function Coverage: 36.73% (8431/22953)
Line Coverage: 29.25% (68590/234486)
Region Coverage: 27.86% (35462/127273)
Branch Coverage: 24.60% (18094/73554)
Coverage Report: http://coverage.selectdb-in.cc/coverage/73dad6dd228881611dcda3a8de8cb4cdc25619d8_73dad6dd228881611dcda3a8de8cb4cdc25619d8/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1808174673
TeamCity be ut coverage result:
Function Coverage: 36.73% (8403/22880)
Line Coverage: 29.27% (68122/232776)
Region Coverage: 27.85% (35167/126261)
Branch Coverage: 24.68% (17975/72830)
Coverage Report: http://coverage.selectdb-in.cc/coverage/00c4a03b98b2e2b0cc739fa453253638d1bb83ab_00c4a03b98b2e2b0cc739fa453253638d1bb83ab/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1810129222
TeamCity be ut coverage result:
Function Coverage: 36.71% (8419/22936)
Line Coverage: 29.25% (68406/233898)
Region Coverage: 27.87% (35357/126858)
Branch Coverage: 24.66% (18067/73256)
Coverage Report: http://coverage.selectdb-in.cc/coverage/004c3dafc11b6f8eb481e12144e73ed32f49e765_004c3dafc11b6f8eb481e12144e73ed32f49e765/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1390859962
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,42 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:100:
```diff
- void add_block_task(PipelineXTask* task);
+ static void add_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,42 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ DCHECK(task->get_state() != PipelineTaskState::RUNNABLE);
+ DCHECK(avoid_using_blocked_queue(task->get_state()));
+ _blocked_task.push_back(task);
+}
+
+void Dependency::_try_to_wake_up_task() {
Review Comment:
warning: method '_try_to_wake_up_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:103:
```diff
- void _try_to_wake_up_task();
+ static void _try_to_wake_up_task();
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +348,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:778:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809634242
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit a453ab0a008e34705d9662570ca4a5a7aa01e65f, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5210 5053 5100 5053
q2 354 161 150 150
q3 2055 1992 1965 1965
q4 1399 1372 1353 1353
q5 3944 3943 3993 3943
q6 253 129 124 124
q7 1476 887 885 885
q8 2785 2773 2756 2756
q9 9572 9536 9340 9340
q10 3448 3528 3531 3528
q11 385 257 257 257
q12 434 283 285 283
q13 4555 4133 4148 4133
q14 320 295 305 295
q15 618 547 546 546
q16 680 587 585 585
q17 1140 1086 1095 1086
q18 8054 7537 7584 7537
q19 1667 1656 1676 1656
q20 542 286 292 286
q21 4661 4343 4305 4305
q22 510 388 403 388
Total cold run time: 54062 ms
Total hot run time: 50454 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5080 5011 5033 5011
q2 328 222 229 222
q3 4059 3993 4018 3993
q4 2796 2757 2758 2757
q5 9769 9798 9936 9798
q6 248 123 126 123
q7 3029 2448 2447 2447
q8 4870 4782 4835 4782
q9 13047 12886 12882 12882
q10 4105 4173 4170 4170
q11 751 657 675 657
q12 993 808 810 808
q13 4250 3853 3870 3853
q14 377 351 361 351
q15 642 575 562 562
q16 751 688 669 669
q17 3859 3870 3843 3843
q18 9685 9263 9260 9260
q19 1827 1755 1758 1755
q20 2384 2056 2043 2043
q21 8906 8871 8606 8606
q22 932 864 849 849
Total cold run time: 82688 ms
Total hot run time: 79441 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809730852
TeamCity be ut coverage result:
Function Coverage: 36.73% (8420/22926)
Line Coverage: 29.26% (68411/233814)
Region Coverage: 27.87% (35353/126838)
Branch Coverage: 24.67% (18071/73250)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b7fe73beee77035f727bb4eaaeffb7c93dcb1908_b7fe73beee77035f727bb4eaaeffb7c93dcb1908/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813783152
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.78 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17158420161 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813749851
TeamCity be ut coverage result:
Function Coverage: 36.76% (8415/22889)
Line Coverage: 29.26% (68396/233724)
Region Coverage: 27.89% (35360/126783)
Branch Coverage: 24.65% (18070/73298)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bc9964f5a59d93490daa9ace5e812c050745d2de_bc9964f5a59d93490daa9ace5e812c050745d2de/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1395180189
##########
be/src/pipeline/pipeline_x/pipeline_x_task.h:
##########
@@ -147,17 +117,78 @@
return _upstream_dependency[id];
}
- Status extract_dependencies();
+ bool is_pipelineX() const override { return true; }
- void push_blocked_task_to_dependency(Dependency* dep) {}
+ void try_wake_up(Dependency* wake_up_dep);
DataSinkOperatorXPtr sink() const { return _sink; }
OperatorXPtr source() const { return _source; }
OperatorXs operatorXs() { return _operators; }
+ bool push_blocked_task_to_queue() {
+ /**
+ * Push task into blocking queue if:
+ * 1. This task is blocked (`_blocked_dep` is not nullptr) and `_use_blocking_queue` is true.
+ * 2. Or this task is blocked by FE two phase execution (BLOCKED_FOR_DEPENDENCY).
+ */
+ return (_blocked_dep && _use_blocking_queue) ||
+ get_state() == PipelineTaskState::BLOCKED_FOR_DEPENDENCY;
+ }
+
private:
+ Dependency* _write_blocked_dependency() {
+ _blocked_dep = _write_dependencies->write_blocked_by(this);
+ if (_blocked_dep != nullptr) {
+ _push_blocked_task_to_dep();
+ static_cast<WriteDependency*>(_blocked_dep)->start_write_watcher();
+ return _blocked_dep;
+ }
+ return nullptr;
+ }
+
+ Dependency* _finish_blocked_dependency(bool skip_current_dep) {
+ for (auto* fin_dep : _finish_dependencies) {
+ if (skip_current_dep && fin_dep == _blocked_dep) {
+ // `_blocked_dep` has already been ready.
+ _blocked_dep = nullptr;
+ continue;
+ }
+ _blocked_dep = fin_dep->finish_blocked_by(this);
+ if (_blocked_dep != nullptr) {
+ _push_blocked_task_to_dep();
+ static_cast<FinishDependency*>(_blocked_dep)->start_finish_watcher();
+ return _blocked_dep;
+ }
+ }
+ return nullptr;
+ }
+
+ Dependency* _read_blocked_dependency() {
Review Comment:
warning: method '_read_blocked_dependency' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Dependency* _read_blocked_dependency() {
```
##########
be/src/pipeline/pipeline_x/pipeline_x_task.h:
##########
@@ -147,17 +117,78 @@ class PipelineXTask : public PipelineTask {
return _upstream_dependency[id];
}
- Status extract_dependencies();
+ bool is_pipelineX() const override { return true; }
- void push_blocked_task_to_dependency(Dependency* dep) {}
+ void try_wake_up(Dependency* wake_up_dep);
DataSinkOperatorXPtr sink() const { return _sink; }
OperatorXPtr source() const { return _source; }
OperatorXs operatorXs() { return _operators; }
+ bool push_blocked_task_to_queue() {
+ /**
+ * Push task into blocking queue if:
+ * 1. This task is blocked (`_blocked_dep` is not nullptr) and `_use_blocking_queue` is true.
+ * 2. Or this task is blocked by FE two phase execution (BLOCKED_FOR_DEPENDENCY).
+ */
+ return (_blocked_dep && _use_blocking_queue) ||
+ get_state() == PipelineTaskState::BLOCKED_FOR_DEPENDENCY;
+ }
+
private:
+ Dependency* _write_blocked_dependency() {
+ _blocked_dep = _write_dependencies->write_blocked_by(this);
+ if (_blocked_dep != nullptr) {
+ _push_blocked_task_to_dep();
+ static_cast<WriteDependency*>(_blocked_dep)->start_write_watcher();
+ return _blocked_dep;
+ }
+ return nullptr;
+ }
+
+ Dependency* _finish_blocked_dependency(bool skip_current_dep) {
Review Comment:
warning: method '_finish_blocked_dependency' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Dependency* _finish_blocked_dependency(bool skip_current_dep) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1397466960
##########
be/src/pipeline/pipeline_task.h:
##########
@@ -251,6 +259,25 @@ class PipelineTask {
void set_parent_profile(RuntimeProfile* profile) { _parent_profile = profile; }
+ virtual bool is_pipelineX() const { return false; }
+
+ void set_blocked(bool blocked) {
+ std::unique_lock<std::mutex> lc(_blocked_lock);
+ if (blocked) {
+ DCHECK(!_blocked) << debug_string();
+ } else {
+ _previous_wake_stack_msg = _wake_stack_msg;
+ _wake_stack_msg = get_stack_trace();
+ DCHECK(_blocked) << debug_string();
+ }
+ _blocked = blocked;
+ }
+
+ void wake_up_by_queue() { _wake_up_by = nullptr; }
Review Comment:
warning: method 'is_blocked' can be made const [readability-make-member-function-const]
```suggestion
bool is_blocked() const {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816729820
TeamCity be ut coverage result:
Function Coverage: 36.75% (8434/22952)
Line Coverage: 29.26% (68612/234480)
Region Coverage: 27.87% (35476/127271)
Branch Coverage: 24.61% (18101/73554)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7ef3945f1c5983b180371bb76c6fc95e38321dfe_7ef3945f1c5983b180371bb76c6fc95e38321dfe/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813986251
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814290266
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815670307
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815928906
TeamCity be ut coverage result:
Function Coverage: 36.71% (8414/22920)
Line Coverage: 29.23% (68427/234086)
Region Coverage: 27.84% (35381/127070)
Branch Coverage: 24.60% (18066/73444)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fd2dddbaea26f715c51072879711b323bc1f7dbc_fd2dddbaea26f715c51072879711b323bc1f7dbc/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1397016890
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +485,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:749:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,176 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_write_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:143:
```diff
- void add_write_block_task(PipelineXTask* task);
+ static void add_write_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,176 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
+ DCHECK(_write_blocked_task.empty() ||
+ _write_blocked_task[_write_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ task->set_blocked(true);
+ _write_blocked_task.push_back(task);
+}
+
+void FinishDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_finish_blocked_task.empty() ||
+ _finish_blocked_task[_finish_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ task->set_blocked(true);
+ _finish_blocked_task.push_back(task);
+}
+
+void RuntimeFilterDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_filter_blocked_task.empty() ||
+ _filter_blocked_task[_filter_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(_blocked_by_rf) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_blocked(true);
+ _filter_blocked_task.push_back(task);
+}
+
+void Dependency::set_ready_for_read() {
+ if (_ready_for_read) {
+ return;
+ }
+ _read_dependency_watcher.stop();
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_read) {
+ return;
+ }
+ _ready_for_read = true;
+ local_block_task.swap(_blocked_task);
+ }
+}
+
+void WriteDependency::set_ready_for_write() {
+ if (_ready_for_write) {
+ return;
+ }
+ _write_dependency_watcher.stop();
+
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_write) {
+ return;
+ }
+ _ready_for_write = true;
+ local_block_task.swap(_write_blocked_task);
+ }
+ for (auto* task : local_block_task) {
+ task->try_wake_up(this);
+ }
+}
+
+void FinishDependency::set_ready_to_finish() {
Review Comment:
warning: method 'set_ready_to_finish' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:173:
```diff
- void set_ready_to_finish();
+ static void set_ready_to_finish();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815655048
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815942653
TeamCity be ut coverage result:
Function Coverage: 36.71% (8414/22919)
Line Coverage: 29.24% (68434/234064)
Region Coverage: 27.85% (35382/127047)
Branch Coverage: 24.61% (18070/73428)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a09d0c5f56a99aa619211c62d08f47406bf3ea7c_a09d0c5f56a99aa619211c62d08f47406bf3ea7c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812091045
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814282927
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812511068
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1394418145
##########
be/src/pipeline/pipeline_task.h:
##########
@@ -251,6 +257,23 @@ class PipelineTask {
void set_parent_profile(RuntimeProfile* profile) { _parent_profile = profile; }
+ virtual bool is_pipelineX() const { return false; }
+
+ void set_blocked(bool blocked) {
+ std::unique_lock<std::mutex> lc(_blocked_lock);
+ if (blocked) {
+ DCHECK(!_blocked) << debug_string();
+ } else {
+ DCHECK(_blocked) << debug_string();
+ }
+ _blocked = blocked;
+ }
+
+ bool is_blocked() {
Review Comment:
warning: method 'is_blocked' can be made const [readability-make-member-function-const]
```suggestion
bool is_blocked() const {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812799603
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815703660
TeamCity be ut coverage result:
Function Coverage: 36.72% (8414/22916)
Line Coverage: 29.25% (68440/234022)
Region Coverage: 27.86% (35374/126993)
Branch Coverage: 24.62% (18072/73398)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e62aa29c85ba86f0c23283fea413dfad2d1fcb55_e62aa29c85ba86f0c23283fea413dfad2d1fcb55/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812925754
TeamCity be ut coverage result:
Function Coverage: 36.76% (8414/22888)
Line Coverage: 29.26% (68392/233746)
Region Coverage: 27.89% (35356/126765)
Branch Coverage: 24.65% (18066/73284)
Coverage Report: http://coverage.selectdb-in.cc/coverage/203f07e7d7ecc417a03ba75c35f550a37c16e9a2_203f07e7d7ecc417a03ba75c35f550a37c16e9a2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812471959
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813708585
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815826822
TeamCity be ut coverage result:
Function Coverage: 36.72% (8415/22917)
Line Coverage: 29.23% (68431/234081)
Region Coverage: 27.85% (35382/127054)
Branch Coverage: 24.61% (18071/73436)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c96e6ebcd5cffdb75ec457ee71bafff95eb28a7e_c96e6ebcd5cffdb75ec457ee71bafff95eb28a7e/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1392085891
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +348,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:779:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809643677
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1391171732
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +348,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:782:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809849112
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1808177035
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit 00c4a03b98b2e2b0cc739fa453253638d1bb83ab, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5296 5070 5182 5070
q2 359 166 160 160
q3 2034 1981 2022 1981
q4 1401 1341 1360 1341
q5 3974 3971 3994 3971
q6 260 133 141 133
q7 1464 884 902 884
q8 2802 2775 2788 2775
q9 9699 9572 9444 9444
q10 3437 3515 3536 3515
q11 391 253 261 253
q12 436 286 289 286
q13 4548 4136 4144 4136
q14 327 284 286 284
q15 661 568 569 568
q16 687 590 590 590
q17 1130 1085 1078 1078
q18 8092 7543 7559 7543
q19 1676 1675 1675 1675
q20 568 320 307 307
q21 4715 4346 4379 4346
q22 522 412 427 412
Total cold run time: 54479 ms
Total hot run time: 50752 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5049 5035 5114 5035
q2 349 223 237 223
q3 4138 4066 3956 3956
q4 2809 2804 2770 2770
q5 9464 9534 9517 9517
q6 245 125 126 125
q7 2750 2335 2330 2330
q8 4813 4835 4819 4819
q9 13012 12924 12888 12888
q10 4058 4134 4141 4134
q11 797 696 692 692
q12 1019 854 822 822
q13 4306 3932 3877 3877
q14 396 349 355 349
q15 631 551 556 551
q16 745 695 705 695
q17 3902 3946 3932 3932
q18 9660 9565 9691 9565
q19 1824 1781 1767 1767
q20 2390 2070 2063 2063
q21 9058 8873 8786 8786
q22 964 885 830 830
Total cold run time: 82379 ms
Total hot run time: 79726 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809592301
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809841466
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit bc47d83080de1b2b497e32c426fc6d22a480e5a6, data reload: false
run tpch-sf100 query with default conf and session variables
q1 5309 5088 5168 5088
q2 378 179 156 156
q3 2044 2062 2048 2048
q4 1425 1370 1351 1351
q5 3956 3969 4008 3969
q6 257 127 128 127
q7 1472 895 904 895
q8 2772 2779 2759 2759
q9 9592 9523 9432 9432
q10 3469 3523 3531 3523
q11 389 255 257 255
q12 433 284 279 279
q13 4559 4113 4114 4113
q14 320 294 298 294
q15 588 557 540 540
q16 677 589 582 582
q17 1129 1084 1064 1064
q18 8045 7480 7621 7480
q19 1662 1683 1674 1674
q20 528 350 296 296
q21 4686 4346 4376 4346
q22 511 405 394 394
Total cold run time: 54201 ms
Total hot run time: 50665 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 5113 4998 5049 4998
q2 320 248 227 227
q3 4016 4043 4021 4021
q4 2798 2799 2770 2770
q5 9735 9780 9685 9685
q6 243 124 123 123
q7 3074 2485 2543 2485
q8 4839 4842 4883 4842
q9 13232 12989 13032 12989
q10 4080 4190 4217 4190
q11 794 651 678 651
q12 1019 816 820 816
q13 4297 3904 3825 3825
q14 402 365 356 356
q15 613 537 539 537
q16 737 695 691 691
q17 3847 3891 3916 3891
q18 9573 9342 9434 9342
q19 1799 1783 1772 1772
q20 2407 2082 2052 2052
q21 8911 8737 8869 8737
q22 959 836 885 836
Total cold run time: 82808 ms
Total hot run time: 79836 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1392282265
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,44 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ DCHECK(task->get_state() != PipelineTaskState::RUNNABLE);
+ DCHECK(avoid_using_blocked_queue(task->get_state()));
+ DCHECK(_blocked_task.empty() || _blocked_task[_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ _blocked_task.push_back(task);
+}
Review Comment:
warning: method '_try_to_wake_up_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:102:
```diff
- void _try_to_wake_up_task();
+ static void _try_to_wake_up_task();
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +350,12 @@
return results;
}
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:779:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809949935
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1396637886
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +449,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:787:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,140 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SOURCE);
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_write_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:158:
```diff
- void add_write_block_task(PipelineXTask* task);
+ static void add_write_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,140 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SOURCE);
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
+ DCHECK(_write_blocked_task.empty() ||
+ _write_blocked_task[_write_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(!_ready_for_write) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SINK);
+ task->set_blocked(true);
+ _write_blocked_task.push_back(task);
+}
+
+void FinishDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_finish_blocked_task.empty() ||
+ _finish_blocked_task[_finish_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(!_ready_to_finish) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_state(PipelineTaskState::PENDING_FINISH);
+ task->set_blocked(true);
+ _finish_blocked_task.push_back(task);
+}
+
+void RuntimeFilterDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_filter_blocked_task.empty() ||
+ _filter_blocked_task[_filter_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(_blocked_by_rf) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_state(PipelineTaskState::BLOCKED_FOR_RF);
+ task->set_blocked(true);
+ _filter_blocked_task.push_back(task);
+}
+
+void Dependency::set_ready_for_read() {
+ if (_ready_for_read) {
+ return;
+ }
+ _read_dependency_watcher.stop();
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_read) {
+ return;
+ }
+ _ready_for_read = true;
+ local_block_task.swap(_blocked_task);
+ }
+}
+
+void WriteDependency::set_ready_for_write() {
+ if (_ready_for_write) {
+ return;
+ }
+ _write_dependency_watcher.stop();
+
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_write) {
+ return;
+ }
+ _ready_for_write = true;
+ local_block_task.swap(_write_blocked_task);
+ }
+ for (auto* task : local_block_task) {
+ DCHECK(task->get_state() != PipelineTaskState::RUNNABLE);
+ DCHECK(avoid_using_blocked_queue(task->get_state()));
+ task->try_wake_up(this);
+ }
+}
+
+void FinishDependency::set_ready_to_finish() {
Review Comment:
warning: method 'set_ready_to_finish' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:194:
```diff
- void set_ready_to_finish();
+ static void set_ready_to_finish();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1395738661
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,140 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SOURCE);
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
+ DCHECK(_write_blocked_task.empty() ||
+ _write_blocked_task[_write_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(!_ready_for_write) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SINK);
+ task->set_blocked(true);
+ _write_blocked_task.push_back(task);
+}
+
+void FinishDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_finish_blocked_task.empty() ||
+ _finish_blocked_task[_finish_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(!_ready_to_finish) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_state(PipelineTaskState::PENDING_FINISH);
+ task->set_blocked(true);
+ _finish_blocked_task.push_back(task);
+}
+
+void RuntimeFilterDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_filter_blocked_task.empty() ||
+ _filter_blocked_task[_filter_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(_blocked_by_rf) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_state(PipelineTaskState::BLOCKED_FOR_RF);
+ task->set_blocked(true);
+ _filter_blocked_task.push_back(task);
+}
+
+void Dependency::set_ready_for_read() {
+ if (_ready_for_read) {
+ return;
+ }
+ _read_dependency_watcher.stop();
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_read) {
+ return;
+ }
+ _ready_for_read = true;
+ local_block_task.swap(_blocked_task);
+ }
+}
+
+void WriteDependency::set_ready_for_write() {
+ if (_ready_for_write) {
+ return;
+ }
+ _write_dependency_watcher.stop();
+
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_write) {
+ return;
+ }
+ _ready_for_write = true;
+ local_block_task.swap(_write_blocked_task);
+ }
+ for (auto* task : local_block_task) {
+ DCHECK(task->get_state() != PipelineTaskState::RUNNABLE);
+ DCHECK(avoid_using_blocked_queue(task->get_state()));
+ task->try_wake_up(this);
+ }
+}
+
+void FinishDependency::set_ready_to_finish() {
Review Comment:
warning: method 'set_ready_to_finish' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:188:
```diff
- void set_ready_to_finish();
+ static void set_ready_to_finish();
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,140 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SOURCE);
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_write_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:152:
```diff
- void add_write_block_task(PipelineXTask* task);
+ static void add_write_block_task(PipelineXTask* task);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816040548
TeamCity be ut coverage result:
Function Coverage: 36.71% (8415/22920)
Line Coverage: 29.23% (68428/234084)
Region Coverage: 27.84% (35371/127069)
Branch Coverage: 24.60% (18067/73444)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5f7e8896e41d1d6f1fde9513ecf5e117d5ca1aa0_5f7e8896e41d1d6f1fde9513ecf5e117d5ca1aa0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816778038
TeamCity be ut coverage result:
Function Coverage: 36.74% (8433/22952)
Line Coverage: 29.26% (68618/234481)
Region Coverage: 27.88% (35479/127271)
Branch Coverage: 24.61% (18102/73554)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d688d4dc89d307821c7cb82019cdc782ad0c3fbe_d688d4dc89d307821c7cb82019cdc782ad0c3fbe/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816152613
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.99 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17099263033 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814645808
TeamCity be ut coverage result:
Function Coverage: 36.74% (8414/22904)
Line Coverage: 29.25% (68422/233886)
Region Coverage: 27.87% (35368/126911)
Branch Coverage: 24.63% (18069/73358)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0afcf51a659456de58c46d78334be51c95871166_0afcf51a659456de58c46d78334be51c95871166/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815770010
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1393691119
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -186,31 +180,27 @@ class FinishDependency final : public Dependency {
return _finish_dependency_watcher.elapsed_time();
}
- [[nodiscard]] FinishDependency* finish_blocked_by() {
- if (config::enable_fuzzy_mode && !_ready_to_finish &&
- _should_log(_finish_dependency_watcher.elapsed_time())) {
- LOG(WARNING) << "========Dependency may be blocked by some reasons: " << name() << " "
- << _node_id;
+ [[nodiscard]] FinishDependency* finish_blocked_by(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (!_ready_to_finish && task) {
+ add_block_task(task);
}
return _ready_to_finish ? nullptr : this;
}
- void set_ready_to_finish() {
- if (_ready_to_finish) {
- return;
- }
- _finish_dependency_watcher.stop();
- _ready_to_finish = true;
- }
-
- void block_finishing() { _ready_to_finish = false; }
+ void set_ready_to_finish();
void* shared_state() override { return nullptr; }
+ std::string debug_string(int indentation_level = 0) override;
+
+ void add_block_task(PipelineXTask* task) override;
protected:
- std::atomic<bool> _ready_to_finish;
+ bool _ready_to_finish;
Review Comment:
warning: use default member initializer for '_ready_to_finish' [modernize-use-default-member-init]
be/src/pipeline/pipeline_x/dependency.h:167:
```diff
- : Dependency(id, node_id, name), _ready_to_finish(true) {}
+ : Dependency(id, node_id, name), {}
```
```suggestion
bool _ready_to_finish{true};
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,135 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SOURCE);
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_write_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:153:
```diff
- void add_write_block_task(PipelineXTask* task);
+ static void add_write_block_task(PipelineXTask* task);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812153066
TeamCity be ut coverage result:
Function Coverage: 36.78% (8422/22897)
Line Coverage: 29.26% (68422/233827)
Region Coverage: 27.88% (35368/126839)
Branch Coverage: 24.66% (18073/73286)
Coverage Report: http://coverage.selectdb-in.cc/coverage/23561196de1bc95f9a8d2578bb16b1cf1656567f_23561196de1bc95f9a8d2578bb16b1cf1656567f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1394176172
##########
be/src/pipeline/pipeline_x/pipeline_x_task.h:
##########
@@ -70,56 +70,66 @@ class PipelineXTask : public PipelineTask {
// must be call after all pipeline task is finish to release resource
Status close(Status exec_status) override;
+ Dependency* read_blocked_dependency() {
+ for (auto* op_dep : _read_dependencies) {
+ _blocked_dep = op_dep->read_blocked_by(this);
+ if (_blocked_dep != nullptr) {
+ // TODO(gabriel):
+ _use_blocking_queue = true;
+ _blocked_dep->start_read_watcher();
+ return _blocked_dep;
+ }
+ }
+ return nullptr;
+ }
+
bool source_can_read() override {
if (_dry_run) {
return true;
}
- for (auto* op_dep : _read_dependencies) {
- auto* dep = op_dep->read_blocked_by();
- if (dep != nullptr) {
- dep->start_read_watcher();
- push_blocked_task_to_dependency(dep);
- return false;
- }
- }
- return true;
+ return read_blocked_dependency() == nullptr;
}
bool runtime_filters_are_ready_or_timeout() override {
- auto* dep = _filter_dependency->filter_blocked_by();
- if (dep != nullptr) {
- push_blocked_task_to_dependency(dep);
- return false;
- }
- return true;
+ throw doris::Exception(ErrorCode::NOT_IMPLEMENTED_ERROR, "Should not reach here!");
+ return false;
}
- bool sink_can_write() override {
- auto* dep = _write_dependencies->write_blocked_by();
- if (dep != nullptr) {
- dep->start_write_watcher();
- push_blocked_task_to_dependency(dep);
- return false;
+ Dependency* write_blocked_dependency() {
+ _blocked_dep = _write_dependencies->write_blocked_by(this);
+ if (_blocked_dep != nullptr) {
+ _push_blocked_task_to_dep();
+ static_cast<WriteDependency*>(_blocked_dep)->start_write_watcher();
+ return _blocked_dep;
}
- return true;
+ return nullptr;
}
+ bool sink_can_write() override { return write_blocked_dependency() == nullptr; }
+
Status finalize() override;
std::string debug_string() override;
- bool is_pending_finish() override {
+ Dependency* finish_blocked_dependency(bool skip_current_dep) {
Review Comment:
warning: method 'finish_blocked_dependency' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Dependency* finish_blocked_dependency(bool skip_current_dep) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812635200
TeamCity be ut coverage result:
Function Coverage: 36.74% (8410/22888)
Line Coverage: 29.24% (68370/233826)
Region Coverage: 27.88% (35345/126790)
Branch Coverage: 24.66% (18069/73278)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d6dba23c7619550ebb7087b060d51bb8190bb32a_d6dba23c7619550ebb7087b060d51bb8190bb32a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809504977
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814532614
TeamCity be ut coverage result:
Function Coverage: 36.74% (8414/22904)
Line Coverage: 29.26% (68429/233886)
Region Coverage: 27.87% (35370/126911)
Branch Coverage: 24.63% (18071/73358)
Coverage Report: http://coverage.selectdb-in.cc/coverage/36a4edd39bc0d82316dd78938627ed907bf07d37_36a4edd39bc0d82316dd78938627ed907bf07d37/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815896187
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1396993216
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,127 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_write_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:150:
```diff
- void add_write_block_task(PipelineXTask* task);
+ static void add_write_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/pipeline_x_task.h:
##########
@@ -147,17 +117,76 @@ class PipelineXTask : public PipelineTask {
return _upstream_dependency[id];
}
- Status extract_dependencies();
+ bool is_pipelineX() const override { return true; }
- void push_blocked_task_to_dependency(Dependency* dep) {}
+ void try_wake_up(Dependency* wake_up_dep);
DataSinkOperatorXPtr sink() const { return _sink; }
OperatorXPtr source() const { return _source; }
OperatorXs operatorXs() { return _operators; }
+ bool push_blocked_task_to_queue() {
+ /**
+ * Push task into blocking queue if:
+ * 1. This task is blocked (`_blocked_dep` is not nullptr) and `_use_blocking_queue` is true.
+ * 2. Or this task is blocked by FE two phase execution (BLOCKED_FOR_DEPENDENCY).
+ */
+ return _use_blocking_queue || get_state() == PipelineTaskState::BLOCKED_FOR_DEPENDENCY;
+ }
+ bool pending_finish() override { return _is_pending_finish; }
+ bool set_is_pending_finish() { return _is_pending_finish = true; }
+
private:
+ Dependency* _write_blocked_dependency() {
+ _blocked_dep = _write_dependencies->write_blocked_by(this);
+ if (_blocked_dep != nullptr) {
+ _push_blocked_task_to_dep();
+ static_cast<WriteDependency*>(_blocked_dep)->start_write_watcher();
+ return _blocked_dep;
+ }
+ return nullptr;
+ }
+
+ Dependency* _finish_blocked_dependency() {
Review Comment:
warning: method '_finish_blocked_dependency' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Dependency* _finish_blocked_dependency() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816105622
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.82 seconds
stream load tsv: 560 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.3 seconds inserted 10000000 Rows, about 353K ops/s
storage size: 17099854825 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816656809
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816955611
TeamCity be ut coverage result:
Function Coverage: 36.75% (8432/22946)
Line Coverage: 29.25% (68582/234470)
Region Coverage: 27.86% (35453/127274)
Branch Coverage: 24.59% (18086/73562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/510092940307e8ae2890dfc646b5444fbc4b3b89_510092940307e8ae2890dfc646b5444fbc4b3b89/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1397509281
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +485,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:750:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815848245
TeamCity be ut coverage result:
Function Coverage: 36.71% (8413/22917)
Line Coverage: 29.22% (68409/234082)
Region Coverage: 27.84% (35372/127056)
Branch Coverage: 24.60% (18067/73438)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e14e71a6ff891ac39fe26251c8944424a9ffd279_e14e71a6ff891ac39fe26251c8944424a9ffd279/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1396838200
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,130 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_fragment_context.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
+ DCHECK(_write_blocked_task.empty() ||
+ _write_blocked_task[_write_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ task->set_blocked(true);
+ _write_blocked_task.push_back(task);
+}
+
+void FinishDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_finish_blocked_task.empty() ||
+ _finish_blocked_task[_finish_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(!_ready_to_finish) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_blocked(true);
+ _finish_blocked_task.push_back(task);
+}
+
+void RuntimeFilterDependency::add_block_task(PipelineXTask* task) {
+ DCHECK(_filter_blocked_task.empty() ||
+ _filter_blocked_task[_filter_blocked_task.size() - 1] != task)
+ << "Duplicate task: " << task->debug_string();
+ DCHECK(_blocked_by_rf) << "It is not allowed: task: " << task->debug_string()
+ << " \n dependency: " << debug_string()
+ << " \n state: " << get_state_name(task->get_state());
+ task->set_blocked(true);
+ _filter_blocked_task.push_back(task);
+}
+
+void Dependency::set_ready_for_read() {
+ if (_ready_for_read) {
+ return;
+ }
+ _read_dependency_watcher.stop();
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_read) {
+ return;
+ }
+ _ready_for_read = true;
+ local_block_task.swap(_blocked_task);
+ }
+}
+
+void WriteDependency::set_ready_for_write() {
+ if (_ready_for_write) {
+ return;
+ }
+ _write_dependency_watcher.stop();
+
+ std::vector<PipelineXTask*> local_block_task {};
+ {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ if (_ready_for_write) {
+ return;
+ }
+ _ready_for_write = true;
+ local_block_task.swap(_write_blocked_task);
+ }
+ for (auto* task : local_block_task) {
+ task->try_wake_up(this);
+ }
+}
+
+void FinishDependency::set_ready_to_finish() {
Review Comment:
warning: method 'set_ready_to_finish' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:189:
```diff
- void set_ready_to_finish();
+ static void set_ready_to_finish();
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +439,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:780:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814474932
TeamCity be ut coverage result:
Function Coverage: 36.72% (8411/22904)
Line Coverage: 29.25% (68406/233885)
Region Coverage: 27.86% (35359/126911)
Branch Coverage: 24.63% (18066/73358)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fce382a704255c7eead16b00bf4e1103815afd3c_fce382a704255c7eead16b00bf4e1103815afd3c/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1812422115
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1390724032
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,44 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
+ DCHECK(task->get_state() != PipelineTaskState::RUNNABLE);
+ DCHECK(avoid_using_blocked_queue(task->get_state()));
+ DCHECK(task->get_state() != PipelineTaskState::BLOCKED_FOR_SINK);
+ _blocked_task.push_back(task);
+}
+
+void Dependency::_try_to_wake_up_task() {
Review Comment:
warning: method '_try_to_wake_up_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:102:
```diff
- void _try_to_wake_up_task();
+ static void _try_to_wake_up_task();
```
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -899,7 +865,30 @@
public:
ENABLE_FACTORY_CREATOR(LocalExchangeSharedState);
std::vector<moodycamel::ConcurrentQueue<PartitionedBlock>> data_queue;
+ std::vector<Dependency*> source_dependencies;
std::atomic<int> running_sink_operators = 0;
+ void add_running_sink_operators() { running_sink_operators++; }
+ void sub_running_sink_operators() {
+ auto val = running_sink_operators.fetch_sub(1);
+ if (val == 1) {
+ _set_ready_for_read();
+ }
+ }
+ void _set_ready_for_read() {
+ for (auto* dep : source_dependencies) {
+ DCHECK(dep);
+ dep->set_ready_for_read();
+ }
+ }
+ void set_dep_by_channel_id(Dependency* dep, int channel_id) {
+ source_dependencies[channel_id] = dep;
+ dep->block_reading();
+ }
+ void set_ready_for_read(int channel_id) {
Review Comment:
warning: method 'set_ready_for_read' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void set_ready_for_read(int channel_id) {
```
##########
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##########
@@ -87,10 +92,36 @@ Status MultiCastDataStreamer::push(RuntimeState* state, doris::vectorized::Block
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+ _set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
Review Comment:
warning: method '_set_ready_for_read' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/multi_cast_data_streamer.h:82:
```diff
- void _set_ready_for_read(int sender_idx);
+ static void _set_ready_for_read(int sender_idx);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,44 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:99:
```diff
- void add_block_task(PipelineXTask* task);
+ static void add_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -899,7 +865,30 @@ struct LocalExchangeSharedState {
public:
ENABLE_FACTORY_CREATOR(LocalExchangeSharedState);
std::vector<moodycamel::ConcurrentQueue<PartitionedBlock>> data_queue;
+ std::vector<Dependency*> source_dependencies;
std::atomic<int> running_sink_operators = 0;
+ void add_running_sink_operators() { running_sink_operators++; }
+ void sub_running_sink_operators() {
+ auto val = running_sink_operators.fetch_sub(1);
+ if (val == 1) {
+ _set_ready_for_read();
+ }
+ }
+ void _set_ready_for_read() {
+ for (auto* dep : source_dependencies) {
+ DCHECK(dep);
+ dep->set_ready_for_read();
+ }
+ }
+ void set_dep_by_channel_id(Dependency* dep, int channel_id) {
Review Comment:
warning: method 'set_dep_by_channel_id' can be made static [readability-convert-member-functions-to-static]
```suggestion
static void set_dep_by_channel_id(Dependency* dep, int channel_id) {
```
##########
be/src/pipeline/exec/multi_cast_data_streamer.cpp:
##########
@@ -87,10 +92,36 @@
for (int i = 0; i < _sender_pos_to_read.size(); ++i) {
if (_sender_pos_to_read[i] == _multi_cast_blocks.end()) {
_sender_pos_to_read[i] = end;
+ _set_ready_for_read(i);
}
}
_eos = eos;
return Status::OK();
}
+void MultiCastDataStreamer::_set_ready_for_read(int sender_idx) {
+ if (_dependencies.empty()) {
+ return;
+ }
+ auto* dep = _dependencies[sender_idx];
+ DCHECK(dep);
+ dep->set_ready_for_read();
+}
+
+void MultiCastDataStreamer::_set_ready_for_read() {
+ for (auto* dep : _dependencies) {
+ DCHECK(dep);
+ dep->set_ready_for_read();
+ }
+}
+
+void MultiCastDataStreamer::_block_reading(int sender_idx) {
Review Comment:
warning: method '_block_reading' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/exec/multi_cast_data_streamer.h:84:
```diff
- void _block_reading(int sender_idx);
+ static void _block_reading(int sender_idx);
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +350,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:777:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
##########
be/src/pipeline/pipeline_x/pipeline_x_task.h:
##########
@@ -70,56 +70,64 @@ class PipelineXTask : public PipelineTask {
// must be call after all pipeline task is finish to release resource
Status close(Status exec_status) override;
+ Dependency* read_blocked_dependency() {
Review Comment:
warning: method 'read_blocked_dependency' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Dependency* read_blocked_dependency() {
```
##########
be/src/pipeline/pipeline_x/pipeline_x_task.h:
##########
@@ -70,56 +70,64 @@
// must be call after all pipeline task is finish to release resource
Status close(Status exec_status) override;
+ Dependency* read_blocked_dependency() {
+ for (auto* op_dep : _read_dependencies) {
+ _blocked_dep = op_dep->read_blocked_by();
+ if (_blocked_dep != nullptr) {
+ _blocked_dep->start_read_watcher();
+ return _blocked_dep;
+ }
+ }
+ return nullptr;
+ }
+
bool source_can_read() override {
if (_dry_run) {
return true;
}
- for (auto* op_dep : _read_dependencies) {
- auto* dep = op_dep->read_blocked_by();
- if (dep != nullptr) {
- dep->start_read_watcher();
- push_blocked_task_to_dependency(dep);
- return false;
- }
+ return read_blocked_dependency() == nullptr;
+ }
+
+ Dependency* filter_blocked_dependency() {
+ _blocked_dep = _filter_dependency->filter_blocked_by();
+ if (_blocked_dep != nullptr) {
+ return _blocked_dep;
}
- return true;
+ return nullptr;
}
bool runtime_filters_are_ready_or_timeout() override {
- auto* dep = _filter_dependency->filter_blocked_by();
- if (dep != nullptr) {
- push_blocked_task_to_dependency(dep);
- return false;
- }
- return true;
+ return filter_blocked_dependency() == nullptr;
}
- bool sink_can_write() override {
- auto* dep = _write_dependencies->write_blocked_by();
- if (dep != nullptr) {
- dep->start_write_watcher();
- push_blocked_task_to_dependency(dep);
- return false;
+ Dependency* write_blocked_dependency() {
+ _blocked_dep = _write_dependencies->write_blocked_by();
+ if (_blocked_dep != nullptr) {
+ static_cast<WriteDependency*>(_blocked_dep)->start_write_watcher();
+ return _blocked_dep;
}
- return true;
+ return nullptr;
}
+ bool sink_can_write() override { return write_blocked_dependency() == nullptr; }
+
Status finalize() override;
std::string debug_string() override;
- bool is_pending_finish() override {
+ Dependency* finish_blocked_dependency() {
Review Comment:
warning: method 'finish_blocked_dependency' can be made static [readability-convert-member-functions-to-static]
```suggestion
static Dependency* finish_blocked_dependency() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1810015664
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813040342
TeamCity be ut coverage result:
Function Coverage: 36.77% (8415/22888)
Line Coverage: 29.26% (68398/233743)
Region Coverage: 27.90% (35362/126765)
Branch Coverage: 24.66% (18074/73284)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3728ada251f28b1d873e32419b8b339d5ceaeaa3_3728ada251f28b1d873e32419b8b339d5ceaeaa3/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1393645774
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -21,10 +21,133 @@
#include <mutex>
#include "common/logging.h"
+#include "pipeline/pipeline_task.h"
+#include "pipeline/pipeline_x/pipeline_x_task.h"
#include "runtime/memory/mem_tracker.h"
namespace doris::pipeline {
+void Dependency::add_block_task(PipelineXTask* task) {
+ // TODO(gabriel): support read dependency
+ if (!_blocked_task.empty() && _blocked_task[_blocked_task.size() - 1] == task) {
+ return;
+ }
+ task->set_state(PipelineTaskState::BLOCKED_FOR_SOURCE);
+ _blocked_task.push_back(task);
+}
+
+void WriteDependency::add_write_block_task(PipelineXTask* task) {
Review Comment:
warning: method 'add_write_block_task' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:159:
```diff
- void add_write_block_task(PipelineXTask* task);
+ static void add_write_block_task(PipelineXTask* task);
```
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -186,31 +186,32 @@ class FinishDependency final : public Dependency {
return _finish_dependency_watcher.elapsed_time();
}
- [[nodiscard]] FinishDependency* finish_blocked_by() {
+ [[nodiscard]] FinishDependency* finish_blocked_by(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
if (config::enable_fuzzy_mode && !_ready_to_finish &&
_should_log(_finish_dependency_watcher.elapsed_time())) {
LOG(WARNING) << "========Dependency may be blocked by some reasons: " << name() << " "
- << _node_id;
+ << _node_id << " block tasks: " << _blocked_task.size();
}
- return _ready_to_finish ? nullptr : this;
- }
-
- void set_ready_to_finish() {
- if (_ready_to_finish) {
- return;
+ if (!_ready_to_finish && task) {
+ add_block_task(task);
}
- _finish_dependency_watcher.stop();
- _ready_to_finish = true;
+ return _ready_to_finish ? nullptr : this;
}
- void block_finishing() { _ready_to_finish = false; }
+ void set_ready_to_finish();
void* shared_state() override { return nullptr; }
+ std::string debug_string(int indentation_level = 0) override;
+
+ void add_block_task(PipelineXTask* task) override;
protected:
- std::atomic<bool> _ready_to_finish;
+ bool _ready_to_finish;
Review Comment:
warning: use default member initializer for '_ready_to_finish' [modernize-use-default-member-init]
be/src/pipeline/pipeline_x/dependency.h:173:
```diff
- : Dependency(id, node_id, name), _ready_to_finish(true) {}
+ : Dependency(id, node_id, name), {}
```
```suggestion
bool _ready_to_finish{true};
```
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +440,12 @@
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:785:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813877812
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.2 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17099071566 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813898288
TeamCity be ut coverage result:
Function Coverage: 36.76% (8414/22889)
Line Coverage: 29.28% (68451/233787)
Region Coverage: 27.89% (35371/126839)
Branch Coverage: 24.64% (18070/73326)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d4a0ba306feec03381bd4f5fe122191694ec8b31_d4a0ba306feec03381bd4f5fe122191694ec8b31/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814087062
TeamCity be ut coverage result:
Function Coverage: 36.74% (8415/22904)
Line Coverage: 29.26% (68446/233889)
Region Coverage: 27.87% (35371/126913)
Branch Coverage: 24.63% (18071/73358)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d000882f9b2ff884f6bf7a3d2d7054c5b9ecacc2_d000882f9b2ff884f6bf7a3d2d7054c5b9ecacc2/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1814105098
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit d000882f9b2ff884f6bf7a3d2d7054c5b9ecacc2, data reload: true
run tpch-sf100 query with default conf and session variables
q1 5520 4610 4645 4610
q2 383 130 129 129
q3 2096 1919 1932 1919
q4 2963 1208 1204 1204
q5 4094 3971 3978 3971
q6 248 119 122 119
q7 1409 868 877 868
q8 4625 2783 2753 2753
q9 10068 9570 9622 9570
q10 9960 3535 3540 3535
q11 379 242 255 242
q12 462 308 288 288
q13 11951 3783 3825 3783
q14 322 282 273 273
q15 582 516 515 515
q16 683 600 593 593
q17 1143 978 942 942
q18 7877 7700 7537 7537
q19 1707 1667 1677 1667
q20 562 310 302 302
q21 4362 3997 3973 3973
q22 461 367 396 367
Total cold run time: 71857 ms
Total hot run time: 49160 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4577 4556 4573 4556
q2 344 237 250 237
q3 4070 4003 4005 4003
q4 2713 2691 2688 2688
q5 9649 9666 9691 9666
q6 241 120 118 118
q7 2666 2298 2362 2298
q8 4497 4493 4508 4493
q9 13230 13625 13113 13113
q10 4098 4165 4183 4165
q11 791 658 663 658
q12 990 822 823 822
q13 4336 3535 3528 3528
q14 384 339 354 339
q15 565 516 514 514
q16 752 686 690 686
q17 3909 3906 3900 3900
q18 9665 9347 8947 8947
q19 1785 1772 1765 1765
q20 2449 2116 2056 2056
q21 8897 8566 8534 8534
q22 918 805 917 805
Total cold run time: 81526 ms
Total hot run time: 77891 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1810065393
TeamCity be ut coverage result:
Function Coverage: 36.72% (8420/22930)
Line Coverage: 29.25% (68404/233835)
Region Coverage: 27.87% (35355/126844)
Branch Coverage: 24.66% (18066/73250)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2b46fbfbc05db2b3e85208b7ff4a95bc28c48677_2b46fbfbc05db2b3e85208b7ff4a95bc28c48677/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809654607
TeamCity be ut coverage result:
Function Coverage: 36.72% (8418/22927)
Line Coverage: 29.25% (68384/233790)
Region Coverage: 27.87% (35333/126798)
Branch Coverage: 24.67% (18064/73222)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a453ab0a008e34705d9662570ca4a5a7aa01e65f_a453ab0a008e34705d9662570ca4a5a7aa01e65f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1392597780
##########
be/src/pipeline/pipeline_x/dependency.cpp:
##########
@@ -294,6 +415,12 @@ std::vector<uint16_t> HashJoinDependency::convert_block_to_null(vectorized::Bloc
return results;
}
+void SetSharedState::set_probe_finished_children(int child_id) {
Review Comment:
warning: method 'set_probe_finished_children' can be made static [readability-convert-member-functions-to-static]
be/src/pipeline/pipeline_x/dependency.h:796:
```diff
- void set_probe_finished_children(int child_id);
+ static void set_probe_finished_children(int child_id);
```
##########
be/src/pipeline/pipeline_x/dependency.h:
##########
@@ -145,72 +146,82 @@ class WriteDependency : public Dependency {
return _write_dependency_watcher.elapsed_time();
}
- [[nodiscard]] virtual WriteDependency* write_blocked_by() {
+ [[nodiscard]] virtual WriteDependency* write_blocked_by(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
if (config::enable_fuzzy_mode && !_ready_for_write &&
_should_log(_write_dependency_watcher.elapsed_time())) {
LOG(WARNING) << "========Dependency may be blocked by some reasons: " << name() << " "
- << id();
+ << id() << " block tasks: " << _blocked_task.size()
+ << "read :" << _ready_for_read << " write :" << _ready_for_write;
+ }
+ if (!_ready_for_write && task) {
+ add_block_task(task);
}
return _ready_for_write ? nullptr : this;
}
- virtual void set_ready_for_write() {
- if (_ready_for_write) {
- return;
- }
- _write_dependency_watcher.stop();
- _ready_for_write = true;
- }
+ virtual void set_ready_for_write();
virtual void block_writing() { _ready_for_write = false; }
+ std::string debug_string(int indentation_level = 0) override;
+ void add_block_task(PipelineXTask* task) override;
+
protected:
+ friend class Dependency;
std::atomic<bool> _ready_for_write;
MonotonicStopWatch _write_dependency_watcher;
+
+private:
+ std::vector<PipelineXTask*> _write_blocked_task;
};
class FinishDependency final : public Dependency {
public:
FinishDependency(int id, int node_id, std::string name)
- : Dependency(id, name), _ready_to_finish(true), _node_id(node_id) {}
+ : Dependency(id, node_id, name), _ready_to_finish(true) {}
~FinishDependency() override = default;
+ void should_finish_after_check() { _ready_to_finish = false; }
void start_finish_watcher() {
for (auto& child : _children) {
((FinishDependency*)child.get())->start_finish_watcher();
}
_finish_dependency_watcher.start();
}
+ bool is_finish_dependency() override { return true; }
+
[[nodiscard]] int64_t finish_watcher_elapse_time() {
return _finish_dependency_watcher.elapsed_time();
}
- [[nodiscard]] FinishDependency* finish_blocked_by() {
+ [[nodiscard]] FinishDependency* finish_blocked_by(PipelineXTask* task) {
+ std::unique_lock<std::mutex> lc(_task_lock);
if (config::enable_fuzzy_mode && !_ready_to_finish &&
_should_log(_finish_dependency_watcher.elapsed_time())) {
LOG(WARNING) << "========Dependency may be blocked by some reasons: " << name() << " "
- << _node_id;
+ << _node_id << " block tasks: " << _blocked_task.size();
}
- return _ready_to_finish ? nullptr : this;
- }
-
- void set_ready_to_finish() {
- if (_ready_to_finish) {
- return;
+ if (!_ready_to_finish && task) {
+ add_block_task(task);
}
- _finish_dependency_watcher.stop();
- _ready_to_finish = true;
+ return _ready_to_finish ? nullptr : this;
}
- void block_finishing() { _ready_to_finish = false; }
+ void set_ready_to_finish();
void* shared_state() override { return nullptr; }
+ std::string debug_string(int indentation_level = 0) override;
+
+ void add_block_task(PipelineXTask* task) override;
protected:
- std::atomic<bool> _ready_to_finish;
+ bool _ready_to_finish;
Review Comment:
warning: use default member initializer for '_ready_to_finish' [modernize-use-default-member-init]
be/src/pipeline/pipeline_x/dependency.h:181:
```diff
- : Dependency(id, node_id, name), _ready_to_finish(true) {}
+ : Dependency(id, node_id, name), {}
```
```suggestion
bool _ready_to_finish{true};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #26879:
URL: https://github.com/apache/doris/pull/26879#discussion_r1396620931
##########
be/src/pipeline/pipeline_task.h:
##########
@@ -251,6 +257,26 @@ class PipelineTask {
void set_parent_profile(RuntimeProfile* profile) { _parent_profile = profile; }
+ virtual bool is_pipelineX() const { return false; }
+
+ void set_blocked(bool blocked) {
+ std::unique_lock<std::mutex> lc(_blocked_lock);
+ if (blocked) {
+ DCHECK(!_blocked) << debug_string();
+ } else {
+ _wake_stack_msg = get_stack_trace();
+ DCHECK(_blocked) << debug_string();
+ }
+ _blocked = blocked;
+ }
+
Review Comment:
warning: method 'is_blocked' can be made const [readability-make-member-function-const]
```suggestion
bool is_blocked() const {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815734104
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.3 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17099076584 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1815925129
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816541778
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816609441
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1816906846
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1807751980
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1809559037
TeamCity be ut coverage result:
Function Coverage: 36.72% (8418/22927)
Line Coverage: 29.25% (68391/233788)
Region Coverage: 27.87% (35338/126801)
Branch Coverage: 24.67% (18063/73224)
Coverage Report: http://coverage.selectdb-in.cc/coverage/63a5e6c475055ca313dde96539ee9554ac0bcc55_63a5e6c475055ca313dde96539ee9554ac0bcc55/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1808037709
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [pipelineX](dependency) Wake up task by dependencies [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #26879:
URL: https://github.com/apache/doris/pull/26879#issuecomment-1813692289
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org