You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Mryange (via GitHub)" <gi...@apache.org> on 2023/12/15 09:41:34 UTC
[PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Mryange opened a new pull request, #28488:
URL: https://github.com/apache/doris/pull/28488
## Proposed changes
core in pipelineX
```
vdata_stream_recvr.h:272] Check failed: _is_cancelled || !_block_queue.empty() || _num_remaining_senders == 0 _is_cancelled: 0, _block_queue_empty: 1, _num_remaining_senders: 2
```
```
bool VDataStreamRecvr::ready_to_read() {
for (const auto& queue : _sender_queues) {
if (queue->should_wait()) {
return false;
}
}
return true;
}
bool VDataStreamRecvr::SenderQueue::should_wait() {
std::unique_lock<std::mutex> l(_lock);
return !_is_cancelled && _block_queue.empty() && _num_remaining_senders > 0;
}
```
However, pipelineX is different, so here it is changed to be consistent.
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1862116999
TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Tpch sf100 test result on commit 5fec3f90bb748c26e9173a8f66a569e5344ce198, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4687 4418 4447 4418
q2 365 170 159 159
q3 1454 1234 1246 1234
q4 1114 916 943 916
q5 3156 3154 3175 3154
q6 241 126 132 126
q7 986 495 479 479
q8 2219 2218 2194 2194
q9 6713 6746 6651 6651
q10 3213 3288 3257 3257
q11 309 183 180 180
q12 352 203 210 203
q13 4548 3750 3815 3750
q14 242 216 216 216
q15 572 522 523 522
q16 435 386 377 377
q17 1019 586 507 507
q18 7221 6986 6958 6958
q19 1562 1467 1407 1407
q20 539 306 283 283
q21 3087 2643 2686 2643
q22 344 282 285 282
Total cold run time: 44378 ms
Total hot run time: 39916 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4389 4334 4328 4328
q2 267 165 175 165
q3 3523 3520 3517 3517
q4 2379 2378 2376 2376
q5 5724 5708 5752 5708
q6 242 119 122 119
q7 2393 1838 1855 1838
q8 3530 3545 3518 3518
q9 9101 9033 9007 9007
q10 3877 3992 3997 3992
q11 478 373 377 373
q12 766 600 611 600
q13 4268 3582 3558 3558
q14 281 260 259 259
q15 566 525 523 523
q16 503 456 481 456
q17 1876 1861 1818 1818
q18 8635 8351 8155 8155
q19 1735 1741 1755 1741
q20 2243 1944 1934 1934
q21 6478 6199 6137 6137
q22 506 430 429 429
Total cold run time: 63760 ms
Total hot run time: 60551 ms
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1862126699
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.44 seconds
stream load tsv: 577 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17221390594 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1857957535
<details>
<summary>TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'</summary>
```
Tpch sf100 test result on commit b11254b5f26a12043339028f425b41b2d06ce0fb, data reload: false
run tpch-sf100 query with default conf and session variables
q1 4714 4438 4417 4417
q2 377 179 158 158
q3 1457 1308 1224 1224
q4 1108 924 887 887
q5 3150 3164 3191 3164
q6 243 128 129 128
q7 1010 497 482 482
q8 2202 2196 2200 2196
q9 6665 6679 6642 6642
q10 3224 3281 3264 3264
q11 326 206 212 206
q12 350 205 211 205
q13 4539 3782 3785 3782
q14 240 218 216 216
q15 572 522 526 522
q16 438 382 389 382
q17 1031 633 598 598
q18 7058 6760 6874 6760
q19 1553 1487 1390 1390
q20 511 327 319 319
q21 3077 2652 2626 2626
q22 346 275 287 275
Total cold run time: 44191 ms
Total hot run time: 39843 ms
run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1 4376 4398 4354 4354
q2 274 162 175 162
q3 3503 3501 3488 3488
q4 2384 2363 2361 2361
q5 5759 5742 5732 5732
q6 236 122 120 120
q7 2385 1852 1879 1852
q8 3512 3537 3531 3531
q9 9013 9000 8965 8965
q10 3904 4021 4009 4009
q11 506 386 389 386
q12 759 589 613 589
q13 4301 3582 3564 3564
q14 286 249 258 249
q15 575 520 528 520
q16 515 462 459 459
q17 1874 1857 1849 1849
q18 8508 8154 8204 8154
q19 1759 1756 1770 1756
q20 2250 1952 1947 1947
q21 6504 6128 6104 6104
q22 483 413 413 413
Total cold run time: 63666 ms
Total hot run time: 60564 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1862068310
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1863940080
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28488:
URL: https://github.com/apache/doris/pull/28488#discussion_r1427773381
##########
be/src/vec/runtime/vdata_stream_recvr.cpp:
##########
@@ -122,6 +122,16 @@ Status VDataStreamRecvr::SenderQueue::_inner_get_batch_without_lock(Block* block
return Status::OK();
}
+void VDataStreamRecvr::SenderQueue::try_set_dep_ready_without_lock() {
Review Comment:
warning: method 'try_set_dep_ready_without_lock' can be made static [readability-convert-member-functions-to-static]
be/src/vec/runtime/vdata_stream_recvr.h:242:
```diff
- void try_set_dep_ready_without_lock();
+ static void try_set_dep_ready_without_lock();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1858035902
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "Mryange (via GitHub)" <gi...@apache.org>.
Mryange commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1857897147
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1857975308
(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.24 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 67 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17219366475 Bytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1863940045
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #28488:
URL: https://github.com/apache/doris/pull/28488#discussion_r1430884304
##########
be/src/vec/runtime/vdata_stream_recvr.cpp:
##########
@@ -122,6 +123,16 @@ Status VDataStreamRecvr::SenderQueue::_inner_get_batch_without_lock(Block* block
return Status::OK();
}
+void VDataStreamRecvr::SenderQueue::try_set_dep_ready_without_lock() {
Review Comment:
warning: method 'try_set_dep_ready_without_lock' can be made static [readability-convert-member-functions-to-static]
be/src/vec/runtime/vdata_stream_recvr.h:243:
```diff
- void try_set_dep_ready_without_lock();
+ static void try_set_dep_ready_without_lock();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #28488:
URL: https://github.com/apache/doris/pull/28488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline. [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #28488:
URL: https://github.com/apache/doris/pull/28488#issuecomment-1862101406
TeamCity be ut coverage result:
Function Coverage: 36.49% (8535/23387)
Line Coverage: 28.63% (69381/242344)
Region Coverage: 27.64% (35894/129852)
Branch Coverage: 24.36% (18333/75248)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5fec3f90bb748c26e9173a8f66a569e5344ce198_5fec3f90bb748c26e9173a8f66a569e5344ce198/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org