You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2024/03/31 03:32:52 UTC
[PR] [improvement](spill) avoid spill if memory is enough [doris]
jacktengg opened a new pull request, #33075:
URL: https://github.com/apache/doris/pull/33075
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545617469
##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -133,13 +133,16 @@ Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized:
SCOPED_TIMER(local_state.exec_time_counter());
RETURN_IF_ERROR(local_state._status);
- RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
+ if (local_state._shared_state->is_spilled) {
+ RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
Review Comment:
This method will merge block again. Do we need clear the previous partition data??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029075563
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545618535
##########
be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp:
##########
@@ -168,13 +168,21 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
auto* runtime_state = local_state._runtime_state.get();
RETURN_IF_ERROR(_agg_sink_operator->sink(runtime_state, in_block, false));
if (eos) {
- LOG(INFO) << "agg node " << id() << " sink eos";
- if (revocable_mem_size(state) > 0) {
- RETURN_IF_ERROR(revoke_memory(state));
- } else {
- for (auto& partition : local_state._shared_state->spill_partitions) {
- RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+ local_state.profile()->add_info_string(
Review Comment:
Not meet EOS.
Shoud add this to profile when the spilled is set to true at first time.
Because we will provide realtime profile in the future, user will find his query is spilled to disk.
If we set it when eos, then user will not find ths spill task when the query not spilled finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028548615
<details>
<summary>TPC-H: <b>Total hot run time: 38742 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a, data reload: false
------ Round 1 ----------------------------------
q1 17635 4155 4063 4063
q2 2114 197 189 189
q3 10464 1270 1386 1270
q4 10189 885 1009 885
q5 7488 2942 2947 2942
q6 216 134 133 133
q7 1130 623 648 623
q8 9422 2064 2036 2036
q9 6770 6207 6138 6138
q10 8470 3508 3517 3508
q11 428 245 248 245
q12 393 226 215 215
q13 17789 2934 2909 2909
q14 282 238 243 238
q15 534 479 479 479
q16 496 385 377 377
q17 963 927 890 890
q18 7766 6347 6483 6347
q19 1608 1521 1518 1518
q20 605 331 309 309
q21 3533 3118 3108 3108
q22 362 320 320 320
Total cold run time: 108657 ms
Total hot run time: 38742 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4091 4022 4019 4019
q2 332 225 223 223
q3 2949 2932 2955 2932
q4 1902 1855 1826 1826
q5 5239 5233 5219 5219
q6 212 126 126 126
q7 2238 1833 1800 1800
q8 3223 3275 3291 3275
q9 8532 8465 8493 8465
q10 3734 4019 3998 3998
q11 569 490 457 457
q12 775 555 584 555
q13 12448 3107 3114 3107
q14 323 311 302 302
q15 532 497 516 497
q16 510 466 437 437
q17 1743 1721 1739 1721
q18 8111 7732 7689 7689
q19 1684 1710 1711 1710
q20 2035 1853 1820 1820
q21 5066 5011 5025 5011
q22 517 445 437 437
Total cold run time: 66765 ms
Total hot run time: 55626 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029068577
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75 with default session variables
Stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 15.5 seconds inserted 10000000 Rows, about 645K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028551815
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
```
Load test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a with default session variables
Stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc: 59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select: 15.7 seconds inserted 10000000 Rows, about 636K ops/s
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545609088
##########
be/src/pipeline/exec/spill_sort_sink_operator.cpp:
##########
@@ -179,9 +179,20 @@ Status SpillSortSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Bloc
local_state._shared_state->in_mem_shared_state->sorter->data_size());
if (eos) {
if (_enable_spill) {
Review Comment:
I think spill_sort_sink_operator is used when _enable_spill== true. So that I think this variable is always true in this case??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545609260
##########
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##########
@@ -783,6 +783,11 @@ Status PartitionedHashJoinProbeOperatorX::get_block(RuntimeState* state, vectori
bool* eos) {
*eos = false;
auto& local_state = get_local_state(state);
+ Defer defer([&] {
+ if (*eos) {
+ LOG(INFO) << "hash probe " << id() << " eos";
Review Comment:
This is a debug message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029051196
<details>
<summary>TPC-H: <b>Total hot run time: 38791 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75, data reload: false
------ Round 1 ----------------------------------
q1 17628 4219 4101 4101
q2 2113 194 180 180
q3 10467 1228 1380 1228
q4 10211 858 964 858
q5 7477 2949 2954 2949
q6 218 131 133 131
q7 1119 628 603 603
q8 9416 2032 2036 2032
q9 6651 6171 6158 6158
q10 8448 3515 3522 3515
q11 426 244 232 232
q12 385 219 217 217
q13 17778 2882 2886 2882
q14 270 239 243 239
q15 541 482 475 475
q16 497 390 385 385
q17 970 906 902 902
q18 7301 6578 6481 6481
q19 1607 1536 1526 1526
q20 616 312 313 312
q21 3549 3159 3087 3087
q22 368 298 312 298
Total cold run time: 108056 ms
Total hot run time: 38791 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4125 4041 4061 4041
q2 334 224 224 224
q3 2979 2960 2962 2960
q4 1900 1866 1864 1864
q5 5226 5222 5229 5222
q6 210 130 131 130
q7 2248 1808 1824 1808
q8 3225 3286 3287 3286
q9 8466 8478 8476 8476
q10 3748 3955 4009 3955
q11 560 475 466 466
q12 769 598 609 598
q13 14184 3067 3119 3067
q14 321 268 267 267
q15 546 488 507 488
q16 533 478 460 460
q17 1783 1749 1754 1749
q18 8334 7699 7868 7699
q19 1696 1684 1688 1684
q20 2043 1812 1848 1812
q21 5136 4965 4910 4910
q22 522 460 453 453
Total cold run time: 68888 ms
Total hot run time: 55619 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029047018
TeamCity be ut coverage result:
Function Coverage: 35.55% (8840/24865)
Line Coverage: 27.29% (72481/265584)
Region Coverage: 26.49% (37509/141606)
Branch Coverage: 23.30% (19125/82092)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a460b67e5ccee77abc0bb31a38fa5896a2650f75_a460b67e5ccee77abc0bb31a38fa5896a2650f75/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545705386
##########
be/src/pipeline/exec/spill_sort_sink_operator.cpp:
##########
@@ -179,9 +179,20 @@ Status SpillSortSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Bloc
local_state._shared_state->in_mem_shared_state->sorter->data_size());
if (eos) {
if (_enable_spill) {
Review Comment:
We currently only support spill for full sort, but when creating the sort operatorX, it's hard to decide whether it's full sort.
It's in the `SortSinkOperatorX::prepare` phase to decide whether to use full sort, so we create
`SpillSortSinkOperatorX` when `enable_sort_spill=true` for all type of sort, but actually only enable spillable sort for full sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028542658
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029066287
<details>
<summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75, data reload: false
query1 0.04 0.04 0.03
query2 0.08 0.04 0.04
query3 0.24 0.05 0.05
query4 1.67 0.07 0.07
query5 0.48 0.49 0.48
query6 1.13 0.66 0.65
query7 0.02 0.01 0.01
query8 0.05 0.04 0.04
query9 0.55 0.51 0.50
query10 0.56 0.56 0.56
query11 0.15 0.11 0.11
query12 0.15 0.12 0.12
query13 0.60 0.59 0.60
query14 0.77 0.77 0.80
query15 0.86 0.83 0.84
query16 0.35 0.35 0.35
query17 0.98 1.01 0.97
query18 0.26 0.24 0.25
query19 1.79 1.71 1.71
query20 0.01 0.00 0.00
query21 15.54 0.75 0.68
query22 3.15 5.32 2.31
query23 17.72 1.25 1.08
query24 1.42 0.22 0.37
query25 0.13 0.10 0.08
query26 0.29 0.18 0.18
query27 0.08 0.09 0.08
query28 13.57 0.95 0.93
query29 12.63 3.36 3.51
query30 0.28 0.08 0.07
query31 2.82 0.38 0.38
query32 3.28 0.47 0.47
query33 2.91 2.81 2.88
query34 15.49 4.30 4.30
query35 4.37 4.38 4.36
query36 0.68 0.48 0.48
query37 0.20 0.17 0.17
query38 0.17 0.17 0.15
query39 0.05 0.04 0.04
query40 0.20 0.15 0.15
query41 0.10 0.05 0.06
query42 0.06 0.05 0.06
query43 0.04 0.04 0.05
Total cold run time: 105.92 s
Total hot run time: 30.56 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028545505
TeamCity be ut coverage result:
Function Coverage: 35.54% (8838/24865)
Line Coverage: 27.28% (72459/265649)
Region Coverage: 26.48% (37496/141594)
Branch Coverage: 23.29% (19117/82080)
Coverage Report: http://coverage.selectdb-in.cc/coverage/877e5f6cdada56bc478e4c19e5e1e2000b81124a_877e5f6cdada56bc478e4c19e5e1e2000b81124a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545690014
##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -133,13 +133,16 @@ Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized:
SCOPED_TIMER(local_state.exec_time_counter());
RETURN_IF_ERROR(local_state._status);
- RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
+ if (local_state._shared_state->is_spilled) {
+ RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
Review Comment:
`initiate_merge_spill_partition_agg_data` will check whether the current partition still has data, if it is, it continue to output data from the current partition; if not, will merge the next partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028541973
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028551371
<details>
<summary>ClickBench: <b>Total hot run time: 30.6 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a, data reload: false
query1 0.04 0.03 0.03
query2 0.07 0.04 0.04
query3 0.25 0.05 0.05
query4 1.66 0.05 0.06
query5 0.52 0.49 0.49
query6 1.13 0.65 0.65
query7 0.02 0.01 0.01
query8 0.05 0.04 0.05
query9 0.56 0.52 0.51
query10 0.56 0.57 0.56
query11 0.16 0.11 0.11
query12 0.14 0.12 0.12
query13 0.61 0.61 0.59
query14 0.77 0.78 0.79
query15 0.86 0.84 0.84
query16 0.34 0.37 0.37
query17 1.00 1.02 0.98
query18 0.25 0.26 0.26
query19 1.87 1.70 1.76
query20 0.01 0.02 0.01
query21 15.55 0.76 0.68
query22 3.56 5.48 2.05
query23 17.58 1.29 1.08
query24 1.51 0.29 0.29
query25 0.14 0.09 0.07
query26 0.28 0.16 0.20
query27 0.08 0.09 0.08
query28 13.47 0.94 0.93
query29 12.54 3.61 3.42
query30 0.28 0.09 0.08
query31 2.79 0.38 0.38
query32 3.30 0.47 0.47
query33 2.83 2.90 2.86
query34 15.56 4.36 4.36
query35 4.38 4.39 4.38
query36 0.68 0.48 0.48
query37 0.21 0.17 0.18
query38 0.16 0.17 0.15
query39 0.04 0.03 0.04
query40 0.18 0.16 0.15
query41 0.10 0.05 0.05
query42 0.06 0.05 0.05
query43 0.05 0.04 0.04
Total cold run time: 106.2 s
Total hot run time: 30.6 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029075588
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545688520
##########
be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp:
##########
@@ -168,13 +168,21 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
auto* runtime_state = local_state._runtime_state.get();
RETURN_IF_ERROR(_agg_sink_operator->sink(runtime_state, in_block, false));
if (eos) {
- LOG(INFO) << "agg node " << id() << " sink eos";
- if (revocable_mem_size(state) > 0) {
- RETURN_IF_ERROR(revoke_memory(state));
- } else {
- for (auto& partition : local_state._shared_state->spill_partitions) {
- RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+ local_state.profile()->add_info_string(
+ "Spilled", local_state._shared_state->is_spilled ? "true" : "false");
+ LOG(INFO) << "agg node " << id()
Review Comment:
If profile is not enabled, we can seach log to check if query is spilled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028550428
<details>
<summary>TPC-DS: <b>Total hot run time: 181873 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a, data reload: false
query1 1242 366 1126 366
query2 6308 2026 1744 1744
query3 6661 222 219 219
query4 24907 21526 21423 21423
query5 4209 421 416 416
query6 293 193 207 193
query7 4608 307 311 307
query8 243 180 187 180
query9 8489 2237 2235 2235
query10 446 255 269 255
query11 14925 14604 14593 14593
query12 152 99 94 94
query13 1636 388 395 388
query14 8515 7028 7003 7003
query15 208 175 179 175
query16 6801 287 277 277
query17 967 586 564 564
query18 1849 280 269 269
query19 204 161 159 159
query20 99 90 96 90
query21 197 133 129 129
query22 4931 4860 4792 4792
query23 33269 32566 32300 32300
query24 11847 3314 3170 3170
query25 603 411 430 411
query26 752 171 167 167
query27 3036 378 386 378
query28 6538 1896 1875 1875
query29 1053 626 627 626
query30 309 166 164 164
query31 1043 795 775 775
query32 101 67 62 62
query33 727 265 263 263
query34 1168 525 534 525
query35 853 745 753 745
query36 1001 874 870 870
query37 139 90 90 90
query38 3515 3548 3559 3548
query39 1640 1607 1615 1607
query40 198 152 148 148
query41 51 48 47 47
query42 132 116 110 110
query43 448 409 408 408
query44 1177 761 752 752
query45 280 270 272 270
query46 1129 845 837 837
query47 2022 1857 1875 1857
query48 406 331 320 320
query49 866 388 380 380
query50 847 443 419 419
query51 6809 6854 6790 6790
query52 121 105 100 100
query53 374 300 303 300
query54 319 271 259 259
query55 100 92 89 89
query56 259 244 257 244
query57 1286 1132 1160 1132
query58 254 243 246 243
query59 2769 2619 2678 2619
query60 267 248 241 241
query61 94 103 90 90
query62 651 470 458 458
query63 314 286 288 286
query64 4545 3490 3265 3265
query65 3039 3032 3023 3023
query66 854 344 337 337
query67 15606 14829 14969 14829
query68 9086 560 582 560
query69 596 336 326 326
query70 1401 1110 1060 1060
query71 515 273 295 273
query72 6647 2570 2416 2416
query73 1563 324 323 323
query74 6771 6251 6282 6251
query75 3716 2304 2314 2304
query76 5621 1153 1182 1153
query77 581 255 251 251
query78 11058 10201 10278 10201
query79 8275 535 524 524
query80 1124 424 435 424
query81 499 218 222 218
query82 378 103 107 103
query83 197 166 165 165
query84 275 90 89 89
query85 839 283 290 283
query86 347 269 306 269
query87 3679 3499 3496 3496
query88 3894 2316 2319 2316
query89 536 370 393 370
query90 1934 178 183 178
query91 140 108 107 107
query92 62 50 52 50
query93 6473 526 525 525
query94 1130 200 191 191
query95 446 342 337 337
query96 618 275 273 273
query97 2650 2481 2523 2481
query98 224 218 221 218
query99 1194 838 849 838
Total cold run time: 293170 ms
Total hot run time: 181873 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545618061
##########
be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp:
##########
@@ -168,13 +168,21 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
auto* runtime_state = local_state._runtime_state.get();
RETURN_IF_ERROR(_agg_sink_operator->sink(runtime_state, in_block, false));
if (eos) {
- LOG(INFO) << "agg node " << id() << " sink eos";
- if (revocable_mem_size(state) > 0) {
- RETURN_IF_ERROR(revoke_memory(state));
- } else {
- for (auto& partition : local_state._shared_state->spill_partitions) {
- RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+ local_state.profile()->add_info_string(
+ "Spilled", local_state._shared_state->is_spilled ? "true" : "false");
+ LOG(INFO) << "agg node " << id()
Review Comment:
This is debug log?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028828643
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545618018
##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -133,13 +133,16 @@ Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized:
SCOPED_TIMER(local_state.exec_time_counter());
RETURN_IF_ERROR(local_state._status);
- RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
+ if (local_state._shared_state->is_spilled) {
+ RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
Review Comment:
why call init merge every time?? It is only called the first time or meet previous partition's EOS?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545609315
##########
be/src/pipeline/exec/spill_sort_sink_operator.cpp:
##########
@@ -179,9 +179,20 @@ Status SpillSortSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Bloc
local_state._shared_state->in_mem_shared_state->sorter->data_size());
if (eos) {
if (_enable_spill) {
- if (revocable_mem_size(state) > 0) {
- RETURN_IF_ERROR(revoke_memory(state));
+ local_state.profile()->add_info_string(
+ "Spilled", local_state._shared_state->is_spilled ? "true" : "false");
+ LOG(INFO) << "sort node " << id()
Review Comment:
This is a debug message?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029021780
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029015741
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #33075:
URL: https://github.com/apache/doris/pull/33075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028541880
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029062226
<details>
<summary>TPC-DS: <b>Total hot run time: 181785 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75, data reload: false
query1 1212 1126 360 360
query2 6333 1908 1862 1862
query3 6673 214 206 206
query4 24961 21483 21459 21459
query5 4171 409 400 400
query6 288 211 177 177
query7 4604 295 320 295
query8 238 179 180 179
query9 8477 2232 2227 2227
query10 449 260 275 260
query11 15006 14555 14699 14555
query12 150 99 95 95
query13 1631 385 390 385
query14 8551 7070 7038 7038
query15 215 180 178 178
query16 6811 281 298 281
query17 952 614 585 585
query18 1852 292 282 282
query19 215 169 175 169
query20 98 94 94 94
query21 195 141 145 141
query22 4979 4832 4740 4740
query23 33394 32521 32471 32471
query24 12616 3226 3196 3196
query25 724 443 443 443
query26 1981 172 175 172
query27 3270 386 405 386
query28 6732 1890 1869 1869
query29 1342 624 625 624
query30 311 154 167 154
query31 1004 747 761 747
query32 100 77 66 66
query33 696 263 269 263
query34 1017 524 528 524
query35 891 757 750 750
query36 1003 880 917 880
query37 271 85 89 85
query38 3804 3656 3572 3572
query39 1669 1623 1613 1613
query40 263 152 143 143
query41 47 45 45 45
query42 117 104 107 104
query43 457 404 416 404
query44 1137 732 731 731
query45 291 270 293 270
query46 1112 829 827 827
query47 1981 1913 1884 1884
query48 413 324 320 320
query49 955 381 378 378
query50 825 436 414 414
query51 6895 6727 6741 6727
query52 122 115 101 101
query53 371 306 313 306
query54 310 237 250 237
query55 93 88 82 82
query56 254 240 238 238
query57 1302 1171 1202 1171
query58 277 246 232 232
query59 2753 2521 2505 2505
query60 264 240 238 238
query61 94 89 88 88
query62 647 441 456 441
query63 317 286 291 286
query64 5784 3620 3169 3169
query65 3128 3050 3069 3050
query66 1358 345 347 345
query67 15235 14878 14720 14720
query68 6992 571 574 571
query69 558 334 336 334
query70 1157 1141 1141 1141
query71 498 282 285 282
query72 6420 2664 2448 2448
query73 783 326 344 326
query74 6833 6386 6286 6286
query75 3425 2304 2315 2304
query76 4435 1123 1223 1123
query77 535 254 252 252
query78 10913 10087 10272 10087
query79 7873 532 524 524
query80 1294 432 404 404
query81 489 224 221 221
query82 1199 104 108 104
query83 197 169 164 164
query84 272 91 88 88
query85 1376 295 284 284
query86 465 274 283 274
query87 3687 3484 3530 3484
query88 4145 2323 2301 2301
query89 578 369 374 369
query90 1946 174 181 174
query91 140 107 105 105
query92 64 52 53 52
query93 6599 516 518 516
query94 1001 193 194 193
query95 422 328 319 319
query96 609 278 269 269
query97 2624 2501 2476 2476
query98 226 210 222 210
query99 1334 865 861 861
Total cold run time: 294777 ms
Total hot run time: 181785 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org