You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2025/04/02 02:41:22 UTC
[PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Gabriel39 opened a new pull request, #49753:
URL: https://github.com/apache/doris/pull/49753
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #49753:
URL: https://github.com/apache/doris/pull/49753
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771202657
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771245754
<details>
<summary>TPC-DS: <b>Total hot run time: 192820 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 90b544c6fa8549e046c1166c45a21691afec051c, data reload: false
query1 1384 1066 1041 1041
query2 6136 1942 1955 1942
query3 11147 4583 4727 4583
query4 25319 23679 22961 22961
query5 4852 635 445 445
query6 303 193 197 193
query7 3981 494 276 276
query8 293 225 215 215
query9 8501 2565 2556 2556
query10 476 314 263 263
query11 15694 15239 14699 14699
query12 165 111 110 110
query13 1571 555 391 391
query14 9634 6157 6209 6157
query15 201 188 171 171
query16 7607 630 481 481
query17 1187 756 610 610
query18 2031 421 323 323
query19 201 202 171 171
query20 141 123 122 122
query21 214 135 115 115
query22 4729 4675 4345 4345
query23 34198 33480 33492 33480
query24 8344 2433 2414 2414
query25 497 498 410 410
query26 1206 280 168 168
query27 2851 502 325 325
query28 4842 2438 2422 2422
query29 714 581 437 437
query30 276 221 194 194
query31 917 884 808 808
query32 77 66 59 59
query33 546 368 317 317
query34 809 882 518 518
query35 830 868 753 753
query36 986 1043 887 887
query37 123 100 80 80
query38 4170 4316 4036 4036
query39 1495 1445 1442 1442
query40 211 119 107 107
query41 52 56 50 50
query42 125 104 114 104
query43 518 537 489 489
query44 1301 808 810 808
query45 185 195 173 173
query46 847 1039 645 645
query47 1856 1915 1814 1814
query48 385 432 312 312
query49 772 503 461 461
query50 672 687 401 401
query51 4275 4293 4203 4203
query52 120 113 96 96
query53 230 256 203 203
query54 600 571 525 525
query55 85 89 88 88
query56 307 308 278 278
query57 1177 1195 1156 1156
query58 266 283 258 258
query59 2744 2867 2790 2790
query60 336 342 309 309
query61 129 130 138 130
query62 765 733 664 664
query63 221 181 178 178
query64 4248 1071 728 728
query65 4457 4360 4384 4360
query66 1334 413 321 321
query67 16749 15717 15478 15478
query68 9692 886 517 517
query69 472 302 269 269
query70 1188 1117 1117 1117
query71 498 337 279 279
query72 5090 4703 4610 4610
query73 697 556 352 352
query74 8930 9101 8744 8744
query75 4413 3214 2777 2777
query76 4330 1180 737 737
query77 1009 362 271 271
query78 9851 10236 9343 9343
query79 2472 802 572 572
query80 620 499 429 429
query81 474 256 220 220
query82 244 131 102 102
query83 260 259 244 244
query84 294 114 86 86
query85 824 357 318 318
query86 330 273 279 273
query87 4507 4614 4535 4535
query88 2821 2225 2212 2212
query89 416 313 280 280
query90 2142 214 217 214
query91 153 149 114 114
query92 77 64 61 61
query93 1944 928 579 579
query94 672 425 305 305
query95 375 286 287 286
query96 491 571 271 271
query97 3104 3289 3156 3156
query98 228 209 196 196
query99 1386 1382 1302 1302
Total cold run time: 283619 ms
Total hot run time: 192820 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771925792
# BE UT Coverage Report
Increment line coverage `45.27% (67/148)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/976f58995ca63ef05709b1fb0c5e6a9962d6ec4f_976f58995ca63ef05709b1fb0c5e6a9962d6ec4f/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/976f58995ca63ef05709b1fb0c5e6a9962d6ec4f_976f58995ca63ef05709b1fb0c5e6a9962d6ec4f/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 52.11% (13951/26774) |
| Line Coverage | 40.77% (119989/294310) |
| Region Coverage | 39.55% (61204/154736) |
| Branch Coverage | 34.21% (30613/89476) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771466432
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771466488
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #49753:
URL: https://github.com/apache/doris/pull/49753#discussion_r2024024901
##########
be/src/pipeline/pipeline_task.h:
##########
@@ -62,7 +63,7 @@ class PipelineTask {
// must be call after all pipeline task is finish to release resource
Status close(Status exec_status, bool close_sink = true);
- PipelineFragmentContext* fragment_context() { return _fragment_context; }
+ std::shared_ptr<PipelineFragmentContext> fragment_context() { return _fragment_context.lock(); }
Review Comment:
最好不要这么写。
这个写法，就得外部的调用者去判断是否是nullptr，而且如果他忘记了判断nullptr，线上就是一个core。
直接返回weak ptr，这样，外部调用者得显示的lock，就能感知到lock 可能会失败。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771234006
<details>
<summary>TPC-H: <b>Total hot run time: 34423 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 90b544c6fa8549e046c1166c45a21691afec051c, data reload: false
------ Round 1 ----------------------------------
q1 26030 5149 5118 5118
q2 2075 273 191 191
q3 10391 1275 681 681
q4 10236 1011 523 523
q5 7528 2350 2368 2350
q6 189 166 135 135
q7 928 739 609 609
q8 9334 1292 1233 1233
q9 7077 5130 5080 5080
q10 6880 2304 1892 1892
q11 481 283 278 278
q12 357 381 222 222
q13 17769 3667 3107 3107
q14 232 229 215 215
q15 546 488 486 486
q16 639 623 582 582
q17 604 864 346 346
q18 7487 7288 7084 7084
q19 1649 997 570 570
q20 340 337 228 228
q21 4652 3538 2499 2499
q22 1055 1036 994 994
Total cold run time: 116479 ms
Total hot run time: 34423 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5311 5166 5169 5166
q2 249 325 232 232
q3 2161 2713 2315 2315
q4 1425 1905 1447 1447
q5 4463 4370 4388 4370
q6 216 180 134 134
q7 1977 1942 1799 1799
q8 2601 2696 2574 2574
q9 7358 7076 7226 7076
q10 2972 3144 2738 2738
q11 589 528 486 486
q12 726 738 642 642
q13 3495 3854 3413 3413
q14 274 286 262 262
q15 512 486 462 462
q16 665 695 651 651
q17 1167 1561 1425 1425
q18 7865 7576 7533 7533
q19 818 837 982 837
q20 1992 2038 1872 1872
q21 5424 4947 4928 4928
q22 1121 1064 1031 1031
Total cold run time: 53381 ms
Total hot run time: 51393 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771450483
<details>
<summary>ClickBench: <b>Total hot run time: 31.45 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ec3df019b45cb47324612c3ee8d9ee5c8a2f2fa, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.11
query3 0.25 0.19 0.19
query4 1.59 0.19 0.18
query5 0.57 0.58 0.58
query6 1.19 0.71 0.72
query7 0.02 0.02 0.02
query8 0.04 0.03 0.03
query9 0.58 0.52 0.51
query10 0.60 0.58 0.57
query11 0.15 0.11 0.12
query12 0.15 0.11 0.12
query13 0.62 0.60 0.60
query14 2.67 2.70 2.82
query15 0.94 0.85 0.85
query16 0.39 0.39 0.37
query17 1.01 1.04 1.03
query18 0.21 0.20 0.20
query19 1.82 1.85 1.92
query20 0.01 0.01 0.01
query21 15.40 0.93 0.54
query22 0.75 1.14 0.61
query23 15.09 1.37 0.63
query24 6.86 2.06 1.10
query25 0.51 0.14 0.06
query26 0.64 0.16 0.13
query27 0.05 0.05 0.06
query28 9.99 0.82 0.42
query29 12.56 3.98 3.30
query30 0.26 0.09 0.07
query31 2.82 0.57 0.38
query32 3.24 0.55 0.47
query33 3.02 3.06 3.09
query34 15.76 5.15 4.52
query35 4.54 4.55 4.55
query36 0.67 0.50 0.48
query37 0.09 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.02 0.02
query40 0.17 0.13 0.14
query41 0.08 0.03 0.03
query42 0.03 0.02 0.03
query43 0.04 0.03 0.02
Total cold run time: 105.63 s
Total hot run time: 31.45 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771634509
<details>
<summary>ClickBench: <b>Total hot run time: 30.87 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 976f58995ca63ef05709b1fb0c5e6a9962d6ec4f, data reload: false
query1 0.04 0.04 0.03
query2 0.13 0.11 0.11
query3 0.26 0.20 0.19
query4 1.58 0.20 0.19
query5 0.60 0.57 0.58
query6 1.19 0.73 0.70
query7 0.02 0.02 0.01
query8 0.04 0.04 0.04
query9 0.59 0.52 0.53
query10 0.57 0.60 0.57
query11 0.16 0.11 0.11
query12 0.15 0.12 0.12
query13 0.62 0.60 0.60
query14 2.72 2.70 2.70
query15 0.94 0.86 0.84
query16 0.38 0.39 0.37
query17 1.07 0.99 1.04
query18 0.22 0.20 0.20
query19 1.94 1.96 1.84
query20 0.01 0.02 0.01
query21 15.43 0.93 0.54
query22 0.75 1.19 0.68
query23 14.93 1.40 0.63
query24 7.52 1.94 0.40
query25 0.46 0.08 0.14
query26 0.72 0.16 0.13
query27 0.05 0.05 0.05
query28 9.64 0.84 0.43
query29 12.55 4.06 3.31
query30 0.24 0.08 0.06
query31 2.82 0.59 0.39
query32 3.23 0.55 0.47
query33 3.01 3.07 3.07
query34 15.77 5.14 4.57
query35 4.52 4.55 4.53
query36 0.67 0.50 0.50
query37 0.08 0.06 0.06
query38 0.05 0.05 0.04
query39 0.03 0.02 0.03
query40 0.17 0.13 0.13
query41 0.08 0.03 0.02
query42 0.04 0.03 0.02
query43 0.04 0.03 0.03
Total cold run time: 106.03 s
Total hot run time: 30.87 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771423182
<details>
<summary>TPC-H: <b>Total hot run time: 34185 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ec3df019b45cb47324612c3ee8d9ee5c8a2f2fa, data reload: false
------ Round 1 ----------------------------------
q1 25790 5248 4974 4974
q2 2058 277 196 196
q3 10382 1278 692 692
q4 10233 1026 539 539
q5 7551 2388 2333 2333
q6 183 168 129 129
q7 903 747 624 624
q8 9327 1295 1123 1123
q9 6835 5206 5087 5087
q10 6850 2301 1909 1909
q11 497 291 273 273
q12 356 357 216 216
q13 17781 3700 3127 3127
q14 230 221 206 206
q15 550 490 522 490
q16 651 611 569 569
q17 594 859 379 379
q18 7550 7190 7164 7164
q19 1712 970 538 538
q20 337 331 228 228
q21 4028 3403 2413 2413
q22 1092 1036 976 976
Total cold run time: 115490 ms
Total hot run time: 34185 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5139 5043 5092 5043
q2 235 318 222 222
q3 2142 2670 2324 2324
q4 1465 1837 1490 1490
q5 4493 4382 4345 4345
q6 216 168 124 124
q7 2043 2063 1808 1808
q8 2569 2557 2561 2557
q9 7312 7105 7228 7105
q10 2995 3139 2750 2750
q11 580 509 504 504
q12 685 763 611 611
q13 3507 3896 3362 3362
q14 277 304 270 270
q15 525 482 488 482
q16 636 722 666 666
q17 1149 1567 1379 1379
q18 7913 7548 7589 7548
q19 817 803 960 803
q20 1934 2006 1871 1871
q21 5353 5039 4936 4936
q22 1181 1129 1070 1070
Total cold run time: 53166 ms
Total hot run time: 51270 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771250901
<details>
<summary>ClickBench: <b>Total hot run time: 31.39 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 90b544c6fa8549e046c1166c45a21691afec051c, data reload: false
query1 0.04 0.04 0.04
query2 0.12 0.11 0.10
query3 0.25 0.20 0.20
query4 1.59 0.20 0.19
query5 0.59 0.58 0.57
query6 1.20 0.72 0.71
query7 0.02 0.02 0.01
query8 0.04 0.04 0.03
query9 0.57 0.52 0.53
query10 0.58 0.59 0.56
query11 0.16 0.11 0.11
query12 0.15 0.12 0.11
query13 0.61 0.61 0.60
query14 2.83 2.70 2.71
query15 0.93 0.86 0.84
query16 0.39 0.39 0.39
query17 1.02 1.01 1.03
query18 0.21 0.20 0.20
query19 1.87 1.91 1.84
query20 0.01 0.01 0.01
query21 15.37 0.92 0.53
query22 0.78 1.13 0.70
query23 14.97 1.37 0.60
query24 7.33 1.46 0.91
query25 0.50 0.24 0.13
query26 0.55 0.16 0.15
query27 0.06 0.05 0.05
query28 9.65 0.80 0.43
query29 12.63 4.02 3.34
query30 0.25 0.09 0.07
query31 2.82 0.60 0.38
query32 3.23 0.55 0.48
query33 3.10 3.04 3.30
query34 15.78 5.13 4.52
query35 4.55 4.50 4.55
query36 0.66 0.49 0.49
query37 0.08 0.06 0.06
query38 0.05 0.04 0.04
query39 0.03 0.03 0.02
query40 0.17 0.14 0.13
query41 0.09 0.03 0.03
query42 0.04 0.02 0.03
query43 0.03 0.03 0.03
Total cold run time: 105.9 s
Total hot run time: 31.39 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #49753:
URL: https://github.com/apache/doris/pull/49753#discussion_r2024023695
##########
be/src/pipeline/pipeline_task.cpp:
##########
@@ -745,11 +753,65 @@ Status PipelineTask::wake_up(Dependency* dep) {
// call by dependency
DCHECK_EQ(_blocked_dep, dep) << "dep : " << dep->debug_string(0) << "task: " << debug_string();
_blocked_dep = nullptr;
+ auto holder = std::dynamic_pointer_cast<PipelineTask>(shared_from_this());
+ DCHECK(_scheduler);
+ _scheduler->erase_blocked_task(&_blocked_iterator);
RETURN_IF_ERROR(_state_transition(PipelineTask::State::RUNNABLE));
- return get_task_queue()->push_back(this);
+ RETURN_IF_ERROR(get_task_queue()->push_back(holder));
+ return Status::OK();
+}
+
+Status PipelineTask::_state_transition(State new_state) {
+ if (_exec_state != new_state) {
+ _state_change_watcher.reset();
+ _state_change_watcher.start();
+ }
+ _task_profile->add_info_string("TaskState", _to_string(new_state));
+ _task_profile->add_info_string("BlockedByDependency", _blocked_dep ? _blocked_dep->name() : "");
+ switch (new_state) {
+ case State::RUNNABLE:
+ if (_exec_state != State::RUNNABLE && _exec_state != State::BLOCKED &&
+ _exec_state != State::INITED) {
+ return Status::InternalError(
+ "Task state transition from {} to {} is not allowed! Task info: {}",
+ _to_string(_exec_state), _to_string(new_state), debug_string());
+ }
+ break;
+ case State::BLOCKED:
+ // If this task is blocked, the blocking dependency will hold the raw pointer of this task.
+ // To ensure this task will not be freed, we use scheduler to hold the shared pointer.
+ DCHECK(_scheduler);
+ _scheduler->hold_blocked_task(shared_from_this(), &_blocked_iterator);
+ if (_exec_state != State::RUNNABLE && _exec_state != State::FINISHED) {
+ return Status::InternalError(
+ "Task state transition from {} to {} is not allowed! Task info: {}",
+ _to_string(_exec_state), _to_string(new_state), debug_string());
+ }
+ break;
+ case State::FINISHED:
+ if (_exec_state != State::RUNNABLE) {
+ return Status::InternalError(
+ "Task state transition from {} to {} is not allowed! Task info: {}",
+ _to_string(_exec_state), _to_string(new_state), debug_string());
+ }
+ break;
+ case State::FINALIZED:
+ if (_exec_state != State::FINISHED && _exec_state != State::INITED) {
+ return Status::InternalError(
+ "Task state transition from {} to {} is not allowed! Task info: {}",
+ _to_string(_exec_state), _to_string(new_state), debug_string());
+ }
+ break;
+ default:
+ return Status::InternalError(
+ "Task state transition from {} to {} is not allowed! Task info: {}",
+ _to_string(_exec_state), _to_string(new_state), debug_string());
+ }
+ _exec_state = new_state;
+ return Status::OK();
}
QueryContext* PipelineTask::query_context() {
- return _fragment_context->get_query_ctx();
+ return _fragment_context.lock()->get_query_ctx();
Review Comment:
如果lock 结果是一个空怎么办？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771595336
<details>
<summary>TPC-H: <b>Total hot run time: 35129 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 976f58995ca63ef05709b1fb0c5e6a9962d6ec4f, data reload: false
------ Round 1 ----------------------------------
q1 25768 5120 5066 5066
q2 2089 284 194 194
q3 10533 1299 689 689
q4 10222 1015 540 540
q5 8368 2402 2322 2322
q6 182 161 132 132
q7 922 745 602 602
q8 9330 1274 1040 1040
q9 7361 5220 5149 5149
q10 6873 2327 1912 1912
q11 481 297 291 291
q12 361 349 224 224
q13 18120 3725 3143 3143
q14 237 238 214 214
q15 546 489 494 489
q16 619 645 579 579
q17 601 851 370 370
q18 8901 7780 7597 7597
q19 6061 2208 878 878
q20 317 321 225 225
q21 3983 3447 2493 2493
q22 1064 1036 980 980
Total cold run time: 122939 ms
Total hot run time: 35129 ms
----- Round 2, with runtime_filter_mode=off -----
q1 5536 5205 5142 5142
q2 233 325 227 227
q3 2328 2812 2472 2472
q4 1563 1943 1530 1530
q5 4518 4378 4408 4378
q6 218 181 128 128
q7 2024 1911 1783 1783
q8 2594 2473 2457 2457
q9 7234 7197 7326 7197
q10 3034 3219 2776 2776
q11 595 504 500 500
q12 732 811 650 650
q13 3720 3967 3403 3403
q14 290 311 270 270
q15 523 481 478 478
q16 656 689 641 641
q17 1169 1537 1388 1388
q18 7686 7565 7224 7224
q19 793 771 854 771
q20 1948 2029 1889 1889
q21 5366 4958 4901 4901
q22 1120 1086 1042 1042
Total cold run time: 53880 ms
Total hot run time: 51247 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771496115
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771621372
<details>
<summary>TPC-DS: <b>Total hot run time: 192533 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 976f58995ca63ef05709b1fb0c5e6a9962d6ec4f, data reload: false
query1 1398 1066 1054 1054
query2 6262 2054 1970 1970
query3 11125 4671 4572 4572
query4 25632 23404 22829 22829
query5 4655 622 449 449
query6 305 208 194 194
query7 3982 494 283 283
query8 287 239 224 224
query9 8529 2606 2584 2584
query10 450 330 264 264
query11 15507 15091 14873 14873
query12 157 110 113 110
query13 1559 514 391 391
query14 8996 6092 6142 6092
query15 204 182 174 174
query16 7423 617 453 453
query17 1170 739 599 599
query18 2008 408 320 320
query19 194 213 160 160
query20 130 129 127 127
query21 206 126 109 109
query22 4485 4686 4417 4417
query23 34660 33345 33250 33250
query24 8365 2474 2454 2454
query25 508 501 433 433
query26 769 284 168 168
query27 2826 518 341 341
query28 4634 2471 2415 2415
query29 640 601 454 454
query30 280 222 200 200
query31 956 860 827 827
query32 73 67 64 64
query33 559 370 310 310
query34 816 879 541 541
query35 813 860 758 758
query36 960 1014 924 924
query37 127 98 90 90
query38 4124 4313 4187 4187
query39 1474 1469 1422 1422
query40 216 121 109 109
query41 53 54 50 50
query42 128 107 103 103
query43 525 525 489 489
query44 1330 848 827 827
query45 184 182 167 167
query46 836 1026 643 643
query47 1850 1917 1814 1814
query48 391 438 301 301
query49 735 520 416 416
query50 662 699 410 410
query51 4233 4339 4260 4260
query52 107 107 98 98
query53 232 258 187 187
query54 588 568 503 503
query55 87 82 85 82
query56 302 307 304 304
query57 1160 1143 1160 1143
query58 273 261 268 261
query59 2726 2979 2757 2757
query60 341 337 308 308
query61 126 155 131 131
query62 766 742 658 658
query63 217 186 183 183
query64 3044 1031 716 716
query65 4338 4214 4256 4214
query66 901 401 308 308
query67 15749 15620 15265 15265
query68 8852 892 568 568
query69 483 313 259 259
query70 1199 1106 1087 1087
query71 479 310 316 310
query72 5270 4726 4613 4613
query73 685 644 344 344
query74 8858 9174 8938 8938
query75 4208 3203 2705 2705
query76 3740 1201 745 745
query77 785 362 290 290
query78 9835 10147 9234 9234
query79 1944 866 573 573
query80 594 524 439 439
query81 484 251 220 220
query82 451 125 95 95
query83 259 261 292 261
query84 245 110 92 92
query85 775 353 308 308
query86 342 313 283 283
query87 4408 4443 4434 4434
query88 3143 2256 2241 2241
query89 398 312 289 289
query90 1995 211 216 211
query91 144 141 121 121
query92 75 65 54 54
query93 1239 978 577 577
query94 701 429 276 276
query95 373 297 283 283
query96 482 571 276 276
query97 3191 3229 3147 3147
query98 223 204 201 201
query99 1418 1375 1266 1266
Total cold run time: 276626 ms
Total hot run time: 192533 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771506334
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771200708
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771359722
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771359264
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771441775
<details>
<summary>TPC-DS: <b>Total hot run time: 193787 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ec3df019b45cb47324612c3ee8d9ee5c8a2f2fa, data reload: false
query1 1379 1064 1060 1060
query2 6175 2006 1965 1965
query3 11093 4645 4659 4645
query4 25634 23679 23171 23171
query5 4950 642 444 444
query6 299 201 188 188
query7 3980 491 282 282
query8 308 245 252 245
query9 8504 2630 2600 2600
query10 501 323 271 271
query11 15523 15058 14945 14945
query12 170 109 109 109
query13 1565 512 404 404
query14 9240 6106 6315 6106
query15 214 188 181 181
query16 7632 608 477 477
query17 1154 768 625 625
query18 2031 414 338 338
query19 205 195 168 168
query20 143 133 121 121
query21 215 134 107 107
query22 4579 4693 4458 4458
query23 34611 33607 33548 33548
query24 8643 2412 2412 2412
query25 504 474 408 408
query26 1190 273 148 148
query27 2823 519 335 335
query28 4682 2429 2446 2429
query29 713 568 440 440
query30 281 232 200 200
query31 937 847 848 847
query32 76 67 65 65
query33 550 374 323 323
query34 809 906 518 518
query35 840 847 809 809
query36 944 1016 888 888
query37 121 125 75 75
query38 4134 4162 4171 4162
query39 1679 1448 1652 1448
query40 223 122 115 115
query41 54 51 52 51
query42 130 112 104 104
query43 510 522 490 490
query44 1310 811 830 811
query45 182 178 172 172
query46 853 1023 647 647
query47 1806 1899 1795 1795
query48 391 426 317 317
query49 755 534 416 416
query50 675 691 416 416
query51 4234 4344 4264 4264
query52 111 110 96 96
query53 229 267 188 188
query54 600 588 525 525
query55 79 84 80 80
query56 308 305 304 304
query57 1171 1163 1131 1131
query58 297 272 280 272
query59 2783 2832 2797 2797
query60 347 328 355 328
query61 131 128 129 128
query62 737 746 681 681
query63 228 192 193 192
query64 4033 1054 707 707
query65 4432 4358 4405 4358
query66 1128 422 320 320
query67 16282 15792 15453 15453
query68 9266 889 518 518
query69 473 298 258 258
query70 1231 1112 1108 1108
query71 486 305 290 290
query72 5673 4719 4721 4719
query73 702 590 348 348
query74 8959 9043 8974 8974
query75 4324 3207 2703 2703
query76 3794 1177 741 741
query77 798 368 284 284
query78 9992 10086 9306 9306
query79 2680 825 556 556
query80 653 507 438 438
query81 509 263 224 224
query82 490 127 98 98
query83 277 252 236 236
query84 292 102 85 85
query85 787 434 309 309
query86 376 266 274 266
query87 4689 4592 4382 4382
query88 3430 2183 2180 2180
query89 406 314 272 272
query90 1889 206 208 206
query91 142 138 111 111
query92 75 61 57 57
query93 2300 933 569 569
query94 659 406 299 299
query95 364 292 324 292
query96 485 566 272 272
query97 3218 3248 3103 3103
query98 246 214 201 201
query99 1432 1399 1259 1259
Total cold run time: 284147 ms
Total hot run time: 193787 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refactor](pipeline) Re-construct ownership of pipeline components [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49753:
URL: https://github.com/apache/doris/pull/49753#issuecomment-2771951346
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org