You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2024/03/31 03:32:52 UTC

[PR] [improvement](spill) avoid spill if memory is enough [doris]

jacktengg opened a new pull request, #33075:
URL: https://github.com/apache/doris/pull/33075

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545617469


##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -133,13 +133,16 @@ Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized:
     SCOPED_TIMER(local_state.exec_time_counter());
     RETURN_IF_ERROR(local_state._status);
 
-    RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
+    if (local_state._shared_state->is_spilled) {
+        RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));

Review Comment:
   This method will merge block again. Do we need clear the previous partition data??



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029075563

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545618535


##########
be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp:
##########
@@ -168,13 +168,21 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
     auto* runtime_state = local_state._runtime_state.get();
     RETURN_IF_ERROR(_agg_sink_operator->sink(runtime_state, in_block, false));
     if (eos) {
-        LOG(INFO) << "agg node " << id() << " sink eos";
-        if (revocable_mem_size(state) > 0) {
-            RETURN_IF_ERROR(revoke_memory(state));
-        } else {
-            for (auto& partition : local_state._shared_state->spill_partitions) {
-                RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+        local_state.profile()->add_info_string(

Review Comment:
   Not meet EOS.
   Shoud add this to profile when the spilled is set to true at first time.
   Because we will provide realtime profile in the future, user will find his query is spilled to disk.
   If we set it when eos, then user will not find ths spill task when the query not spilled finished.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028548615

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38742 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17635	4155	4063	4063
   q2	2114	197	189	189
   q3	10464	1270	1386	1270
   q4	10189	885	1009	885
   q5	7488	2942	2947	2942
   q6	216	134	133	133
   q7	1130	623	648	623
   q8	9422	2064	2036	2036
   q9	6770	6207	6138	6138
   q10	8470	3508	3517	3508
   q11	428	245	248	245
   q12	393	226	215	215
   q13	17789	2934	2909	2909
   q14	282	238	243	238
   q15	534	479	479	479
   q16	496	385	377	377
   q17	963	927	890	890
   q18	7766	6347	6483	6347
   q19	1608	1521	1518	1518
   q20	605	331	309	309
   q21	3533	3118	3108	3108
   q22	362	320	320	320
   Total cold run time: 108657 ms
   Total hot run time: 38742 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4091	4022	4019	4019
   q2	332	225	223	223
   q3	2949	2932	2955	2932
   q4	1902	1855	1826	1826
   q5	5239	5233	5219	5219
   q6	212	126	126	126
   q7	2238	1833	1800	1800
   q8	3223	3275	3291	3275
   q9	8532	8465	8493	8465
   q10	3734	4019	3998	3998
   q11	569	490	457	457
   q12	775	555	584	555
   q13	12448	3107	3114	3107
   q14	323	311	302	302
   q15	532	497	516	497
   q16	510	466	437	437
   q17	1743	1721	1739	1721
   q18	8111	7732	7689	7689
   q19	1684	1710	1711	1710
   q20	2035	1853	1820	1820
   q21	5066	5011	5025	5011
   q22	517	445	437	437
   Total cold run time: 66765 ms
   Total hot run time: 55626 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029068577

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       15.5 seconds inserted 10000000 Rows, about 645K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028551815

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       15.7 seconds inserted 10000000 Rows, about 636K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545609088


##########
be/src/pipeline/exec/spill_sort_sink_operator.cpp:
##########
@@ -179,9 +179,20 @@ Status SpillSortSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Bloc
             local_state._shared_state->in_mem_shared_state->sorter->data_size());
     if (eos) {
         if (_enable_spill) {

Review Comment:
   I think   spill_sort_sink_operator is used when _enable_spill== true. So that I think this variable is always true in this case??



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545609260


##########
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##########
@@ -783,6 +783,11 @@ Status PartitionedHashJoinProbeOperatorX::get_block(RuntimeState* state, vectori
                                                     bool* eos) {
     *eos = false;
     auto& local_state = get_local_state(state);
+    Defer defer([&] {
+        if (*eos) {
+            LOG(INFO) << "hash probe " << id() << " eos";

Review Comment:
   This is a debug message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029051196

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38791 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17628	4219	4101	4101
   q2	2113	194	180	180
   q3	10467	1228	1380	1228
   q4	10211	858	964	858
   q5	7477	2949	2954	2949
   q6	218	131	133	131
   q7	1119	628	603	603
   q8	9416	2032	2036	2032
   q9	6651	6171	6158	6158
   q10	8448	3515	3522	3515
   q11	426	244	232	232
   q12	385	219	217	217
   q13	17778	2882	2886	2882
   q14	270	239	243	239
   q15	541	482	475	475
   q16	497	390	385	385
   q17	970	906	902	902
   q18	7301	6578	6481	6481
   q19	1607	1536	1526	1526
   q20	616	312	313	312
   q21	3549	3159	3087	3087
   q22	368	298	312	298
   Total cold run time: 108056 ms
   Total hot run time: 38791 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4125	4041	4061	4041
   q2	334	224	224	224
   q3	2979	2960	2962	2960
   q4	1900	1866	1864	1864
   q5	5226	5222	5229	5222
   q6	210	130	131	130
   q7	2248	1808	1824	1808
   q8	3225	3286	3287	3286
   q9	8466	8478	8476	8476
   q10	3748	3955	4009	3955
   q11	560	475	466	466
   q12	769	598	609	598
   q13	14184	3067	3119	3067
   q14	321	268	267	267
   q15	546	488	507	488
   q16	533	478	460	460
   q17	1783	1749	1754	1749
   q18	8334	7699	7868	7699
   q19	1696	1684	1688	1684
   q20	2043	1812	1848	1812
   q21	5136	4965	4910	4910
   q22	522	460	453	453
   Total cold run time: 68888 ms
   Total hot run time: 55619 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029047018

   TeamCity be ut coverage result:
    Function Coverage: 35.55% (8840/24865) 
    Line Coverage: 27.29% (72481/265584)
    Region Coverage: 26.49% (37509/141606)
    Branch Coverage: 23.30% (19125/82092)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/a460b67e5ccee77abc0bb31a38fa5896a2650f75_a460b67e5ccee77abc0bb31a38fa5896a2650f75/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545705386


##########
be/src/pipeline/exec/spill_sort_sink_operator.cpp:
##########
@@ -179,9 +179,20 @@ Status SpillSortSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Bloc
             local_state._shared_state->in_mem_shared_state->sorter->data_size());
     if (eos) {
         if (_enable_spill) {

Review Comment:
   We currently only support spill for full sort, but when creating the sort operatorX, it's hard to decide whether it's full sort.
   
   It's in the `SortSinkOperatorX::prepare` phase to decide whether to use full sort, so we create 
    `SpillSortSinkOperatorX` when `enable_sort_spill=true` for all type of sort, but actually only enable spillable sort for full sort.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028542658

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029066287

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.56 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75, data reload: false
   
   query1	0.04	0.04	0.03
   query2	0.08	0.04	0.04
   query3	0.24	0.05	0.05
   query4	1.67	0.07	0.07
   query5	0.48	0.49	0.48
   query6	1.13	0.66	0.65
   query7	0.02	0.01	0.01
   query8	0.05	0.04	0.04
   query9	0.55	0.51	0.50
   query10	0.56	0.56	0.56
   query11	0.15	0.11	0.11
   query12	0.15	0.12	0.12
   query13	0.60	0.59	0.60
   query14	0.77	0.77	0.80
   query15	0.86	0.83	0.84
   query16	0.35	0.35	0.35
   query17	0.98	1.01	0.97
   query18	0.26	0.24	0.25
   query19	1.79	1.71	1.71
   query20	0.01	0.00	0.00
   query21	15.54	0.75	0.68
   query22	3.15	5.32	2.31
   query23	17.72	1.25	1.08
   query24	1.42	0.22	0.37
   query25	0.13	0.10	0.08
   query26	0.29	0.18	0.18
   query27	0.08	0.09	0.08
   query28	13.57	0.95	0.93
   query29	12.63	3.36	3.51
   query30	0.28	0.08	0.07
   query31	2.82	0.38	0.38
   query32	3.28	0.47	0.47
   query33	2.91	2.81	2.88
   query34	15.49	4.30	4.30
   query35	4.37	4.38	4.36
   query36	0.68	0.48	0.48
   query37	0.20	0.17	0.17
   query38	0.17	0.17	0.15
   query39	0.05	0.04	0.04
   query40	0.20	0.15	0.15
   query41	0.10	0.05	0.06
   query42	0.06	0.05	0.06
   query43	0.04	0.04	0.05
   Total cold run time: 105.92 s
   Total hot run time: 30.56 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028545505

   TeamCity be ut coverage result:
    Function Coverage: 35.54% (8838/24865) 
    Line Coverage: 27.28% (72459/265649)
    Region Coverage: 26.48% (37496/141594)
    Branch Coverage: 23.29% (19117/82080)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/877e5f6cdada56bc478e4c19e5e1e2000b81124a_877e5f6cdada56bc478e4c19e5e1e2000b81124a/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545690014


##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -133,13 +133,16 @@ Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized:
     SCOPED_TIMER(local_state.exec_time_counter());
     RETURN_IF_ERROR(local_state._status);
 
-    RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
+    if (local_state._shared_state->is_spilled) {
+        RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));

Review Comment:
   `initiate_merge_spill_partition_agg_data` will check whether the current partition still has data, if it is, it continue to output data from the current partition; if not, will merge the next partition.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028541973

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028551371

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.6 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.07	0.04	0.04
   query3	0.25	0.05	0.05
   query4	1.66	0.05	0.06
   query5	0.52	0.49	0.49
   query6	1.13	0.65	0.65
   query7	0.02	0.01	0.01
   query8	0.05	0.04	0.05
   query9	0.56	0.52	0.51
   query10	0.56	0.57	0.56
   query11	0.16	0.11	0.11
   query12	0.14	0.12	0.12
   query13	0.61	0.61	0.59
   query14	0.77	0.78	0.79
   query15	0.86	0.84	0.84
   query16	0.34	0.37	0.37
   query17	1.00	1.02	0.98
   query18	0.25	0.26	0.26
   query19	1.87	1.70	1.76
   query20	0.01	0.02	0.01
   query21	15.55	0.76	0.68
   query22	3.56	5.48	2.05
   query23	17.58	1.29	1.08
   query24	1.51	0.29	0.29
   query25	0.14	0.09	0.07
   query26	0.28	0.16	0.20
   query27	0.08	0.09	0.08
   query28	13.47	0.94	0.93
   query29	12.54	3.61	3.42
   query30	0.28	0.09	0.08
   query31	2.79	0.38	0.38
   query32	3.30	0.47	0.47
   query33	2.83	2.90	2.86
   query34	15.56	4.36	4.36
   query35	4.38	4.39	4.38
   query36	0.68	0.48	0.48
   query37	0.21	0.17	0.18
   query38	0.16	0.17	0.15
   query39	0.04	0.03	0.04
   query40	0.18	0.16	0.15
   query41	0.10	0.05	0.05
   query42	0.06	0.05	0.05
   query43	0.05	0.04	0.04
   Total cold run time: 106.2 s
   Total hot run time: 30.6 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029075588

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545688520


##########
be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp:
##########
@@ -168,13 +168,21 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
     auto* runtime_state = local_state._runtime_state.get();
     RETURN_IF_ERROR(_agg_sink_operator->sink(runtime_state, in_block, false));
     if (eos) {
-        LOG(INFO) << "agg node " << id() << " sink eos";
-        if (revocable_mem_size(state) > 0) {
-            RETURN_IF_ERROR(revoke_memory(state));
-        } else {
-            for (auto& partition : local_state._shared_state->spill_partitions) {
-                RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+        local_state.profile()->add_info_string(
+                "Spilled", local_state._shared_state->is_spilled ? "true" : "false");
+        LOG(INFO) << "agg node " << id()

Review Comment:
   If profile is not enabled, we can seach log to check if query is spilled.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028550428

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 181873 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 877e5f6cdada56bc478e4c19e5e1e2000b81124a, data reload: false
   
   query1	1242	366	1126	366
   query2	6308	2026	1744	1744
   query3	6661	222	219	219
   query4	24907	21526	21423	21423
   query5	4209	421	416	416
   query6	293	193	207	193
   query7	4608	307	311	307
   query8	243	180	187	180
   query9	8489	2237	2235	2235
   query10	446	255	269	255
   query11	14925	14604	14593	14593
   query12	152	99	94	94
   query13	1636	388	395	388
   query14	8515	7028	7003	7003
   query15	208	175	179	175
   query16	6801	287	277	277
   query17	967	586	564	564
   query18	1849	280	269	269
   query19	204	161	159	159
   query20	99	90	96	90
   query21	197	133	129	129
   query22	4931	4860	4792	4792
   query23	33269	32566	32300	32300
   query24	11847	3314	3170	3170
   query25	603	411	430	411
   query26	752	171	167	167
   query27	3036	378	386	378
   query28	6538	1896	1875	1875
   query29	1053	626	627	626
   query30	309	166	164	164
   query31	1043	795	775	775
   query32	101	67	62	62
   query33	727	265	263	263
   query34	1168	525	534	525
   query35	853	745	753	745
   query36	1001	874	870	870
   query37	139	90	90	90
   query38	3515	3548	3559	3548
   query39	1640	1607	1615	1607
   query40	198	152	148	148
   query41	51	48	47	47
   query42	132	116	110	110
   query43	448	409	408	408
   query44	1177	761	752	752
   query45	280	270	272	270
   query46	1129	845	837	837
   query47	2022	1857	1875	1857
   query48	406	331	320	320
   query49	866	388	380	380
   query50	847	443	419	419
   query51	6809	6854	6790	6790
   query52	121	105	100	100
   query53	374	300	303	300
   query54	319	271	259	259
   query55	100	92	89	89
   query56	259	244	257	244
   query57	1286	1132	1160	1132
   query58	254	243	246	243
   query59	2769	2619	2678	2619
   query60	267	248	241	241
   query61	94	103	90	90
   query62	651	470	458	458
   query63	314	286	288	286
   query64	4545	3490	3265	3265
   query65	3039	3032	3023	3023
   query66	854	344	337	337
   query67	15606	14829	14969	14829
   query68	9086	560	582	560
   query69	596	336	326	326
   query70	1401	1110	1060	1060
   query71	515	273	295	273
   query72	6647	2570	2416	2416
   query73	1563	324	323	323
   query74	6771	6251	6282	6251
   query75	3716	2304	2314	2304
   query76	5621	1153	1182	1153
   query77	581	255	251	251
   query78	11058	10201	10278	10201
   query79	8275	535	524	524
   query80	1124	424	435	424
   query81	499	218	222	218
   query82	378	103	107	103
   query83	197	166	165	165
   query84	275	90	89	89
   query85	839	283	290	283
   query86	347	269	306	269
   query87	3679	3499	3496	3496
   query88	3894	2316	2319	2316
   query89	536	370	393	370
   query90	1934	178	183	178
   query91	140	108	107	107
   query92	62	50	52	50
   query93	6473	526	525	525
   query94	1130	200	191	191
   query95	446	342	337	337
   query96	618	275	273	273
   query97	2650	2481	2523	2481
   query98	224	218	221	218
   query99	1194	838	849	838
   Total cold run time: 293170 ms
   Total hot run time: 181873 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545618061


##########
be/src/pipeline/exec/partitioned_aggregation_sink_operator.cpp:
##########
@@ -168,13 +168,21 @@ Status PartitionedAggSinkOperatorX::sink(doris::RuntimeState* state, vectorized:
     auto* runtime_state = local_state._runtime_state.get();
     RETURN_IF_ERROR(_agg_sink_operator->sink(runtime_state, in_block, false));
     if (eos) {
-        LOG(INFO) << "agg node " << id() << " sink eos";
-        if (revocable_mem_size(state) > 0) {
-            RETURN_IF_ERROR(revoke_memory(state));
-        } else {
-            for (auto& partition : local_state._shared_state->spill_partitions) {
-                RETURN_IF_ERROR(partition->finish_current_spilling(eos));
+        local_state.profile()->add_info_string(
+                "Spilled", local_state._shared_state->is_spilled ? "true" : "false");
+        LOG(INFO) << "agg node " << id()

Review Comment:
   This is debug log?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028828643

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545618018


##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -133,13 +133,16 @@ Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized:
     SCOPED_TIMER(local_state.exec_time_counter());
     RETURN_IF_ERROR(local_state._status);
 
-    RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));
+    if (local_state._shared_state->is_spilled) {
+        RETURN_IF_ERROR(local_state.initiate_merge_spill_partition_agg_data(state));

Review Comment:
   why call init merge every time?? It is only called the first time or meet previous partition's EOS?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #33075:
URL: https://github.com/apache/doris/pull/33075#discussion_r1545609315


##########
be/src/pipeline/exec/spill_sort_sink_operator.cpp:
##########
@@ -179,9 +179,20 @@ Status SpillSortSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Bloc
             local_state._shared_state->in_mem_shared_state->sorter->data_size());
     if (eos) {
         if (_enable_spill) {
-            if (revocable_mem_size(state) > 0) {
-                RETURN_IF_ERROR(revoke_memory(state));
+            local_state.profile()->add_info_string(
+                    "Spilled", local_state._shared_state->is_spilled ? "true" : "false");
+            LOG(INFO) << "sort node " << id()

Review Comment:
   This is a debug message?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029021780

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029015741

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #33075:
URL: https://github.com/apache/doris/pull/33075


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2028541880

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) avoid spill if memory is enough [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #33075:
URL: https://github.com/apache/doris/pull/33075#issuecomment-2029062226

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 181785 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit a460b67e5ccee77abc0bb31a38fa5896a2650f75, data reload: false
   
   query1	1212	1126	360	360
   query2	6333	1908	1862	1862
   query3	6673	214	206	206
   query4	24961	21483	21459	21459
   query5	4171	409	400	400
   query6	288	211	177	177
   query7	4604	295	320	295
   query8	238	179	180	179
   query9	8477	2232	2227	2227
   query10	449	260	275	260
   query11	15006	14555	14699	14555
   query12	150	99	95	95
   query13	1631	385	390	385
   query14	8551	7070	7038	7038
   query15	215	180	178	178
   query16	6811	281	298	281
   query17	952	614	585	585
   query18	1852	292	282	282
   query19	215	169	175	169
   query20	98	94	94	94
   query21	195	141	145	141
   query22	4979	4832	4740	4740
   query23	33394	32521	32471	32471
   query24	12616	3226	3196	3196
   query25	724	443	443	443
   query26	1981	172	175	172
   query27	3270	386	405	386
   query28	6732	1890	1869	1869
   query29	1342	624	625	624
   query30	311	154	167	154
   query31	1004	747	761	747
   query32	100	77	66	66
   query33	696	263	269	263
   query34	1017	524	528	524
   query35	891	757	750	750
   query36	1003	880	917	880
   query37	271	85	89	85
   query38	3804	3656	3572	3572
   query39	1669	1623	1613	1613
   query40	263	152	143	143
   query41	47	45	45	45
   query42	117	104	107	104
   query43	457	404	416	404
   query44	1137	732	731	731
   query45	291	270	293	270
   query46	1112	829	827	827
   query47	1981	1913	1884	1884
   query48	413	324	320	320
   query49	955	381	378	378
   query50	825	436	414	414
   query51	6895	6727	6741	6727
   query52	122	115	101	101
   query53	371	306	313	306
   query54	310	237	250	237
   query55	93	88	82	82
   query56	254	240	238	238
   query57	1302	1171	1202	1171
   query58	277	246	232	232
   query59	2753	2521	2505	2505
   query60	264	240	238	238
   query61	94	89	88	88
   query62	647	441	456	441
   query63	317	286	291	286
   query64	5784	3620	3169	3169
   query65	3128	3050	3069	3050
   query66	1358	345	347	345
   query67	15235	14878	14720	14720
   query68	6992	571	574	571
   query69	558	334	336	334
   query70	1157	1141	1141	1141
   query71	498	282	285	282
   query72	6420	2664	2448	2448
   query73	783	326	344	326
   query74	6833	6386	6286	6286
   query75	3425	2304	2315	2304
   query76	4435	1123	1223	1123
   query77	535	254	252	252
   query78	10913	10087	10272	10087
   query79	7873	532	524	524
   query80	1294	432	404	404
   query81	489	224	221	221
   query82	1199	104	108	104
   query83	197	169	164	164
   query84	272	91	88	88
   query85	1376	295	284	284
   query86	465	274	283	274
   query87	3687	3484	3530	3484
   query88	4145	2323	2301	2301
   query89	578	369	374	369
   query90	1946	174	181	174
   query91	140	107	105	105
   query92	64	52	53	52
   query93	6599	516	518	516
   query94	1001	193	194	193
   query95	422	328	319	319
   query96	609	278	269	269
   query97	2624	2501	2476	2476
   query98	226	210	222	210
   query99	1334	865	861	861
   Total cold run time: 294777 ms
   Total hot run time: 181785 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org