You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2024/03/21 13:01:07 UTC

[PR] [improvement](spill) spill trigger improvement [doris]

jacktengg opened a new pull request, #32641:
URL: https://github.com/apache/doris/pull/32641

   ## Proposed changes
   
   Issue Number: close #xxx
   
   <!--Describe your changes.-->
   
   ## Further comments
   
   If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017726728

   TeamCity be ut coverage result:
    Function Coverage: 35.23% (8733/24790) 
    Line Coverage: 27.03% (71513/264560)
    Region Coverage: 26.28% (37103/141203)
    Branch Coverage: 23.17% (18971/81894)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/0e279b3fac84fecc43a2e6fc9829bd07ba245b4f_0e279b3fac84fecc43a2e6fc9829bd07ba245b4f/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2012234980

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542289829


##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +82,31 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
 
     int64_t memory_used();
 
+    int spill_threshold_low_water_mark() const {
+        return _spill_low_watermark.load(std::memory_order_relaxed);
+    }
+    int spill_threashold_high_water_mark() const {
+        return _spill_high_watermark.load(std::memory_order_relaxed);
+    }
+
+    void set_weighted_memory_used(int64_t wg_total_mem_used, double ratio);
+
+    std::unordered_set<std::shared_ptr<MemTrackerLimiter>> get_all_mem_trackers();
+
+    int64_t get_weighted_memory_used() {
+        return _weighted_mem_used.load(std::memory_order_relaxed);
+    }
+
+    void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {

Review Comment:
   add const to the function signature.
   And do not use reference as return value. Use pointer instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1533875471


##########
be/src/pipeline/exec/partitioned_hash_join_sink_operator.cpp:
##########
@@ -48,15 +48,25 @@ Status PartitionedHashJoinSinkLocalState::open(RuntimeState* state) {
     RETURN_IF_ERROR(PipelineXSinkLocalState::open(state));
     return _partitioner->open(state);
 }
+Status PartitionedHashJoinSinkLocalState::close(RuntimeState* state, Status exec_status) {

Review Comment:
   warning: method 'close' can be made static [readability-convert-member-functions-to-static]
   
   be/src/pipeline/exec/partitioned_hash_join_sink_operator.h:48:
   ```diff
   -     Status close(RuntimeState* state, Status exec_status) override;
   +     static Status close(RuntimeState* state, Status exec_status) override;
   ```
   



##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +82,24 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
 
     int64_t memory_used();
 
+    double spill_threshold_low_water_mark() const { return _spill_threshold_low_water_mark; }
+    double spill_threashold_high_water_mark() const { return _spill_threashold_high_water_mark; }
+
+    void set_weighted_memory_used(int64_t tg_total_mem_used, double ratio);
+
+    std::unordered_set<std::shared_ptr<MemTrackerLimiter>> get_all_mem_trackers();
+
+    int64_t get_weighted_memory_used() {
+        return _weighted_mem_used.load(std::memory_order_relaxed);
+    }
+
+    void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {

Review Comment:
   warning: method 'check_mem_used' can be made const [readability-make-member-function-const]
   
   ```suggestion
       void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) const {
   ```
   



##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_used_memory() {

Review Comment:
   warning: function 'refresh_wg_used_memory' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   void WorkloadGroupMgr::refresh_wg_used_memory() {
                          ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **be/src/runtime/workload_group/workload_group_manager.cpp:146:** 106 lines including whitespace and comments (threshold 80)
   ```cpp
   void WorkloadGroupMgr::refresh_wg_used_memory() {
                          ^
   ```
   
   </details>
   



##########
be/src/runtime/memory/mem_tracker_limiter.h:
##########
@@ -244,6 +245,20 @@ class MemTrackerLimiter final : public MemTracker {
     // Log the memory usage when memory limit is exceeded.
     std::string tracker_limit_exceeded_str();
 
+    void set_weighted_mem(int64_t weighted_limit, double ratio) {
+        std::lock_guard<std::mutex> l(_weighted_mem_lock);
+        _weighted_consumption = _consumption->current_value() * ratio;
+        _weighted_limit = weighted_limit;
+    }
+    void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) {

Review Comment:
   warning: method 'get_weighted_mem_info' can be made const [readability-make-member-function-const]
   
   ```suggestion
       void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) const {
   ```
   



##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_used_memory() {

Review Comment:
   warning: method 'refresh_wg_used_memory' can be made static [readability-convert-member-functions-to-static]
   
   be/src/runtime/workload_group/workload_group_manager.h:56:
   ```diff
   -     void refresh_wg_used_memory();
   +     static void refresh_wg_used_memory();
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1540421187


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,125 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {

Review Comment:
   warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **be/src/runtime/workload_group/workload_group_manager.cpp:146:** 110 lines including whitespace and comments (threshold 80)
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026780194

   TeamCity be ut coverage result:
    Function Coverage: 35.56% (8839/24858) 
    Line Coverage: 27.29% (72455/265547)
    Region Coverage: 26.51% (37528/141569)
    Branch Coverage: 23.31% (19129/82066)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/c8f6c475195c6fa5512777393f6824eb63ce1a44_c8f6c475195c6fa5512777393f6824eb63ce1a44/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024330793

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 181686 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 615fff13fc8d598fd9aad875a0fb730eabc282d0, data reload: false
   
   query1	927	365	355	355
   query2	6535	2044	1843	1843
   query3	6700	215	207	207
   query4	31645	21303	21350	21303
   query5	4419	395	400	395
   query6	269	188	172	172
   query7	4634	297	294	294
   query8	227	167	165	165
   query9	9350	2287	2275	2275
   query10	567	252	246	246
   query11	17232	14208	14192	14192
   query12	139	94	87	87
   query13	1626	414	429	414
   query14	9618	7643	7571	7571
   query15	243	207	199	199
   query16	8158	271	281	271
   query17	1938	579	539	539
   query18	2103	281	269	269
   query19	283	148	147	147
   query20	96	83	85	83
   query21	199	129	123	123
   query22	5141	4820	4812	4812
   query23	33463	32680	33122	32680
   query24	10744	2839	2819	2819
   query25	586	364	357	357
   query26	1219	150	151	150
   query27	3002	353	356	353
   query28	7598	1901	1862	1862
   query29	856	614	604	604
   query30	293	147	146	146
   query31	941	714	737	714
   query32	94	56	52	52
   query33	760	256	252	252
   query34	1062	480	498	480
   query35	818	620	604	604
   query36	1003	911	901	901
   query37	124	66	66	66
   query38	3560	3415	3421	3415
   query39	1468	1440	1436	1436
   query40	227	113	111	111
   query41	47	45	45	45
   query42	102	92	95	92
   query43	492	461	458	458
   query44	1206	732	727	727
   query45	264	262	273	262
   query46	1114	715	694	694
   query47	1889	1840	1840	1840
   query48	436	359	355	355
   query49	1119	328	331	328
   query50	753	364	365	364
   query51	6662	6619	6597	6597
   query52	95	90	89	89
   query53	342	270	270	270
   query54	284	230	230	230
   query55	84	80	78	78
   query56	243	216	214	214
   query57	1214	1148	1130	1130
   query58	227	203	199	199
   query59	2800	2548	2591	2548
   query60	258	232	239	232
   query61	100	95	95	95
   query62	655	453	457	453
   query63	316	276	276	276
   query64	5725	4202	4149	4149
   query65	3059	3027	3051	3027
   query66	949	386	369	369
   query67	15297	14930	14908	14908
   query68	5281	520	520	520
   query69	568	377	371	371
   query70	1246	1161	1177	1161
   query71	405	271	266	266
   query72	6510	2865	2682	2682
   query73	714	316	324	316
   query74	7221	6495	6542	6495
   query75	2988	2266	2239	2239
   query76	3483	847	859	847
   query77	389	271	257	257
   query78	10964	10142	10146	10142
   query79	8327	515	517	515
   query80	1896	395	408	395
   query81	557	223	222	222
   query82	1580	86	89	86
   query83	307	155	149	149
   query84	293	82	82	82
   query85	1731	317	312	312
   query86	496	307	289	289
   query87	3723	3478	3590	3478
   query88	5224	2383	2284	2284
   query89	522	362	377	362
   query90	1943	176	178	176
   query91	171	138	137	137
   query92	66	50	47	47
   query93	6118	486	485	485
   query94	1257	178	174	174
   query95	444	333	336	333
   query96	587	266	269	266
   query97	2689	2497	2473	2473
   query98	227	213	208	208
   query99	1200	935	924	924
   Total cold run time: 304726 ms
   Total hot run time: 181686 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542711431


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,127 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {

Review Comment:
   warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **be/src/runtime/workload_group/workload_group_manager.cpp:146:** 112 lines including whitespace and comments (threshold 80)
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   
   </details>
   



##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,48 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    auto wg = query_ctx->workload_group();
+    if (!wg) {
+        LOG_ONCE(INFO) << "no workload group for query " << print_id(state->query_id());
+        return false;
+    }
+    bool is_wg_mem_low_water_mark = false;
+    bool is_wg_mem_high_water_mark = false;
+    wg->check_mem_used(&is_wg_mem_low_water_mark, &is_wg_mem_high_water_mark);
+    if (is_wg_mem_high_water_mark) {
+        if (revocable_mem_bytes > 0) {
+            LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+            return true;

Review Comment:
   warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
   
   be/src/pipeline/pipeline_x/pipeline_x_task.cpp:317:
   ```diff
   -         if (revocable_mem_bytes > 0) {
   -             LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
   -             return true;
   -         }
   -         return false;
   +         return revocable_mem_bytes > 0;
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026677979

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026766666

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37970 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17638	4315	4162	4162
   q2	2111	155	146	146
   q3	10593	1139	1216	1139
   q4	10226	786	818	786
   q5	7469	2976	2913	2913
   q6	205	124	128	124
   q7	1038	584	578	578
   q8	9328	2024	2028	2024
   q9	6977	6395	6364	6364
   q10	8422	3461	3549	3461
   q11	419	230	218	218
   q12	425	205	190	190
   q13	17808	2841	2864	2841
   q14	227	214	208	208
   q15	509	479	461	461
   q16	474	363	367	363
   q17	954	530	622	530
   q18	7192	6663	6546	6546
   q19	1659	1462	1430	1430
   q20	573	268	266	266
   q21	3686	2936	2922	2922
   q22	342	298	318	298
   Total cold run time: 108275 ms
   Total hot run time: 37970 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4169	4110	4098	4098
   q2	327	232	226	226
   q3	2955	2859	2865	2859
   q4	1885	1538	1615	1538
   q5	5202	5227	5240	5227
   q6	199	119	119	119
   q7	2192	1826	1818	1818
   q8	3175	3340	3303	3303
   q9	8478	8443	8452	8443
   q10	3820	3858	3950	3858
   q11	586	467	465	465
   q12	757	575	579	575
   q13	17028	3034	3082	3034
   q14	292	270	279	270
   q15	540	479	478	478
   q16	494	466	449	449
   q17	1793	1530	1550	1530
   q18	8135	7797	7655	7655
   q19	2884	1553	1588	1553
   q20	2010	1883	1702	1702
   q21	4947	4784	4885	4784
   q22	557	476	486	476
   Total cold run time: 72425 ms
   Total hot run time: 54460 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026787628

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       13.9 seconds inserted 10000000 Rows, about 719K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024333754

   TeamCity be ut coverage result:
    Function Coverage: 35.22% (8741/24817) 
    Line Coverage: 27.00% (71549/265007)
    Region Coverage: 26.25% (37121/141401)
    Branch Coverage: 23.15% (18981/81990)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/615fff13fc8d598fd9aad875a0fb730eabc282d0_615fff13fc8d598fd9aad875a0fb730eabc282d0/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1544065493


##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -327,6 +305,50 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    auto wg = query_ctx->workload_group();
+    if (!wg) {
+        LOG_ONCE(INFO) << "no workload group for query " << print_id(state->query_id());
+        return false;
+    }
+    bool is_wg_mem_low_water_mark = false;
+    bool is_wg_mem_high_water_mark = false;
+    wg->check_mem_used(&is_wg_mem_low_water_mark, &is_wg_mem_high_water_mark);
+    if (is_wg_mem_high_water_mark) {
+        if (revocable_mem_bytes > 0) {
+            LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+            return true;

Review Comment:
   warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
   
   be/src/pipeline/pipeline_x/pipeline_x_task.cpp:318:
   ```diff
   -         if (revocable_mem_bytes > 0) {
   -             LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
   -             return true;
   -         }
   -         return false;
   +         return revocable_mem_bytes > 0;
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021937403

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 181926 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit e1783354448196d335626f926218759a3286ec2e, data reload: false
   
   query1	926	369	352	352
   query2	6547	1890	1840	1840
   query3	6701	213	215	213
   query4	31457	21417	21308	21308
   query5	4318	413	400	400
   query6	272	181	192	181
   query7	4624	288	285	285
   query8	229	169	171	169
   query9	9277	2307	2277	2277
   query10	557	243	249	243
   query11	15277	14252	14268	14252
   query12	135	90	93	90
   query13	1623	415	429	415
   query14	10233	7626	8040	7626
   query15	243	204	191	191
   query16	8233	266	262	262
   query17	1960	591	556	556
   query18	2104	301	286	286
   query19	359	160	161	160
   query20	100	89	88	88
   query21	204	128	131	128
   query22	5101	4849	4798	4798
   query23	33344	32743	32746	32743
   query24	10814	2879	2875	2875
   query25	627	388	385	385
   query26	1398	157	161	157
   query27	2957	357	357	357
   query28	7618	1913	1894	1894
   query29	909	642	634	634
   query30	302	156	149	149
   query31	976	728	752	728
   query32	86	57	57	57
   query33	770	260	250	250
   query34	1049	485	490	485
   query35	847	609	605	605
   query36	1013	871	887	871
   query37	130	64	69	64
   query38	3530	3394	3466	3394
   query39	1466	1458	1460	1458
   query40	205	111	114	111
   query41	48	47	46	46
   query42	104	95	94	94
   query43	485	440	447	440
   query44	1203	724	730	724
   query45	253	265	264	264
   query46	1108	748	711	711
   query47	1935	1839	1836	1836
   query48	447	387	347	347
   query49	1101	330	332	330
   query50	764	374	372	372
   query51	6699	6631	6595	6595
   query52	102	96	91	91
   query53	351	272	271	271
   query54	289	228	240	228
   query55	87	85	83	83
   query56	245	222	219	219
   query57	1227	1141	1128	1128
   query58	235	206	201	201
   query59	2698	2567	2639	2567
   query60	267	238	242	238
   query61	96	91	91	91
   query62	658	463	441	441
   query63	303	281	274	274
   query64	5804	4034	3993	3993
   query65	3130	3047	3021	3021
   query66	1425	372	347	347
   query67	15263	15201	14903	14903
   query68	5359	521	525	521
   query69	574	374	362	362
   query70	1196	1153	1071	1071
   query71	405	264	272	264
   query72	6343	2885	2675	2675
   query73	703	316	320	316
   query74	7020	6571	6513	6513
   query75	2950	2266	2214	2214
   query76	3545	911	908	908
   query77	418	271	266	266
   query78	10855	10307	10127	10127
   query79	8706	527	528	527
   query80	2120	400	395	395
   query81	560	220	217	217
   query82	1385	93	86	86
   query83	290	149	149	149
   query84	293	84	84	84
   query85	2195	323	314	314
   query86	502	287	293	287
   query87	3702	3572	3548	3548
   query88	4931	2322	2314	2314
   query89	553	384	373	373
   query90	1993	181	179	179
   query91	169	140	139	139
   query92	66	47	47	47
   query93	6534	504	492	492
   query94	1243	181	180	180
   query95	439	331	334	331
   query96	601	274	278	274
   query97	2675	2498	2473	2473
   query98	239	225	207	207
   query99	1240	900	881	881
   Total cold run time: 304652 ms
   Total hot run time: 181926 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542643876


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+    std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+    // workload group id -> workload group queries
+    std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+            all_wg_queries;
+    for (auto& [wg_id, wg] : _workload_groups) {
+        all_wg_queries.insert({wg_id, wg->queries()});
+    }
+
+    int64_t all_queries_mem_used = 0;
+
+    // calculate total memory used of each workload group and total memory used of all queries
+    std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+    for (auto& [wg_id, wg_queries] : all_wg_queries) {
+        int64_t wg_total_mem_used = 0;
+        for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+            if (auto query_ctx = query_ctx_ptr.lock()) {
+                wg_total_mem_used +=
+                        query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+            }
+        }
+        all_queries_mem_used += wg_total_mem_used;
+        wgs_mem_info[wg_id] = {wg_total_mem_used};
+    }
+
+    auto proc_vm_rss = PerfCounters::get_vm_rss();
+    if (all_queries_mem_used <= 0) {
+        return;
+    }
+
+    auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+    auto sys_mem_available = doris::MemInfo::sys_mem_available();
+    std::string debug_msg = fmt::format(
+            "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+            "{}, all quries mem: {}",
+            PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+            PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+            PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+            PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+    LOG_EVERY_N(INFO, 5) << debug_msg;
+
+    if (proc_vm_rss < all_queries_mem_used) {
+        all_queries_mem_used = proc_vm_rss;
+    }
+
+    // process memory used is actually bigger than all_queries_mem_used,
+    // because memory of page cache, allocator cache, segment cache etc. are included
+    // in process_mem_used.
+    // we count these cache memories equally on workload groups.
+    double ratio = (double)proc_vm_rss / (double)all_queries_mem_used;
+
+    for (auto& wg : _workload_groups) {
+        auto wg_mem_limit = wg.second->memory_limit();
+        auto& wg_mem_info = wgs_mem_info[wg.first];
+        wg_mem_info.weighted_mem_used = wg_mem_info.total_mem_used * ratio;
+        wg_mem_info.mem_used_ratio = (double)wg_mem_info.weighted_mem_used / wg_mem_limit;
+
+        wg.second->set_weighted_memory_used(wg_mem_info.total_mem_used, ratio);
+
+        auto spill_low_water_mark = wg.second->spill_threshold_low_water_mark();
+        auto spill_high_water_mark = wg.second->spill_threashold_high_water_mark();
+        wg_mem_info.is_high_wartermark = (wg_mem_info.weighted_mem_used >
+                                          ((double)wg_mem_limit * spill_high_water_mark / 100));
+        wg_mem_info.is_low_wartermark = (wg_mem_info.weighted_mem_used >
+                                         ((double)wg_mem_limit * spill_low_water_mark / 100));
+
+        // calculate query weighted memory limit of task group
+        const auto& wg_queries = all_wg_queries[wg.first];
+        auto wg_query_count = wg_queries.size();

Review Comment:
   这里如果query count  < 1 感觉可以直接continue了;
   那么216 行直接就是wg_mem_limit + wg_query_count) / wg_query_count



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024972418

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37769 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit df0ddde4d7f8ba7db99389a85bb281abe6306fe4, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17620	4264	4122	4122
   q2	2109	160	155	155
   q3	10779	1174	1271	1174
   q4	10697	771	761	761
   q5	7467	2966	2924	2924
   q6	203	126	125	125
   q7	942	536	497	497
   q8	9357	2072	2012	2012
   q9	6944	6367	6326	6326
   q10	8470	3444	3586	3444
   q11	435	236	223	223
   q12	388	209	200	200
   q13	17814	2911	2881	2881
   q14	233	193	205	193
   q15	508	475	466	466
   q16	495	384	373	373
   q17	968	521	608	521
   q18	7230	6589	6508	6508
   q19	2381	1449	1515	1449
   q20	547	275	264	264
   q21	3598	2995	2859	2859
   q22	362	292	309	292
   Total cold run time: 109547 ms
   Total hot run time: 37769 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4177	4061	4062	4061
   q2	329	227	227	227
   q3	2993	2835	2823	2823
   q4	1856	1600	1571	1571
   q5	5238	5269	5257	5257
   q6	198	116	119	116
   q7	1753	1369	1367	1367
   q8	3197	3312	3324	3312
   q9	8518	8441	8525	8441
   q10	3784	3848	3770	3770
   q11	547	449	446	446
   q12	723	565	540	540
   q13	16924	2836	2865	2836
   q14	275	248	243	243
   q15	503	474	460	460
   q16	488	429	413	413
   q17	1749	1511	1468	1468
   q18	7427	7128	7085	7085
   q19	1613	1491	1557	1491
   q20	1905	1721	1721	1721
   q21	4874	4729	4655	4655
   q22	518	444	444	444
   Total cold run time: 69589 ms
   Total hot run time: 52747 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024321329

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37958 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 615fff13fc8d598fd9aad875a0fb730eabc282d0, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17601	4755	4141	4141
   q2	2108	154	157	154
   q3	10654	1158	1223	1158
   q4	10232	829	772	772
   q5	7453	3067	2958	2958
   q6	200	123	124	123
   q7	1101	594	572	572
   q8	9355	2018	2030	2018
   q9	7370	6689	6609	6609
   q10	8462	3488	3563	3488
   q11	438	224	213	213
   q12	425	193	193	193
   q13	17789	2892	2854	2854
   q14	241	200	208	200
   q15	512	462	465	462
   q16	503	378	387	378
   q17	959	565	611	565
   q18	7086	6574	6358	6358
   q19	3989	1378	1430	1378
   q20	555	258	262	258
   q21	3548	2813	2990	2813
   q22	365	293	300	293
   Total cold run time: 110946 ms
   Total hot run time: 37958 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4147	4124	4041	4041
   q2	332	233	227	227
   q3	3082	2880	2847	2847
   q4	1875	1552	1561	1552
   q5	5302	5367	5330	5330
   q6	195	117	117	117
   q7	2225	1849	1853	1849
   q8	3190	3319	3317	3317
   q9	8716	8702	8670	8670
   q10	3825	3787	3777	3777
   q11	530	452	438	438
   q12	711	560	550	550
   q13	16928	2867	2856	2856
   q14	285	249	254	249
   q15	501	467	464	464
   q16	484	423	419	419
   q17	1740	1527	1471	1471
   q18	7471	7148	7000	7000
   q19	1624	1547	1548	1547
   q20	1911	1739	1709	1709
   q21	4936	4770	4693	4693
   q22	542	427	460	427
   Total cold run time: 70552 ms
   Total hot run time: 53550 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542288795


##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+    if (!need_revoke) {
+        return false;
+    }
+
+    const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+    auto tg = query_ctx->workload_group();

Review Comment:
   If tg == nullptr then return false; to avoid core when there is no workload group bind to this query. And print log in this case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542289053


##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+    if (!need_revoke) {
+        return false;
+    }
+
+    const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+    auto tg = query_ctx->workload_group();

Review Comment:
   do not use tg. tg is old name "task group", use wg



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2025006677

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit df0ddde4d7f8ba7db99389a85bb281abe6306fe4 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       14.0 seconds inserted 10000000 Rows, about 714K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021868148

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024860541

   TeamCity be ut coverage result:
    Function Coverage: 35.58% (8832/24825) 
    Line Coverage: 27.29% (72377/265175)
    Region Coverage: 26.53% (37532/141479)
    Branch Coverage: 23.34% (19145/82032)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/09207c21e85cb9bb7da144b2960060fec2a8e348_09207c21e85cb9bb7da144b2960060fec2a8e348/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026784311

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 29.21 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44, data reload: false
   
   query1	0.04	0.03	0.03
   query2	0.06	0.02	0.02
   query3	0.22	0.06	0.06
   query4	1.65	0.09	0.10
   query5	0.50	0.48	0.47
   query6	1.14	0.66	0.63
   query7	0.01	0.01	0.02
   query8	0.04	0.02	0.03
   query9	0.55	0.50	0.50
   query10	0.54	0.54	0.55
   query11	0.12	0.09	0.09
   query12	0.11	0.09	0.09
   query13	0.58	0.58	0.59
   query14	0.74	0.76	0.77
   query15	0.81	0.78	0.79
   query16	0.36	0.40	0.37
   query17	0.92	1.04	0.95
   query18	0.21	0.22	0.24
   query19	1.79	1.75	1.68
   query20	0.01	0.02	0.02
   query21	15.54	0.57	0.56
   query22	2.68	2.73	1.15
   query23	17.13	0.84	0.73
   query24	3.32	2.16	1.38
   query25	0.19	0.14	0.20
   query26	0.63	0.14	0.12
   query27	0.04	0.03	0.03
   query28	10.43	0.86	0.83
   query29	12.52	3.24	3.20
   query30	0.27	0.09	0.08
   query31	2.82	0.36	0.36
   query32	3.33	0.43	0.45
   query33	2.77	2.78	2.84
   query34	16.16	4.38	4.32
   query35	4.41	4.43	4.45
   query36	0.66	0.47	0.46
   query37	0.08	0.06	0.06
   query38	0.05	0.03	0.04
   query39	0.04	0.02	0.02
   query40	0.17	0.13	0.14
   query41	0.07	0.02	0.02
   query42	0.02	0.02	0.02
   query43	0.02	0.02	0.03
   Total cold run time: 103.75 s
   Total hot run time: 29.21 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024866370

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 37716 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 09207c21e85cb9bb7da144b2960060fec2a8e348, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17613	4536	4120	4120
   q2	2108	159	151	151
   q3	10587	1176	1207	1176
   q4	10242	831	803	803
   q5	7470	2941	2940	2940
   q6	200	124	127	124
   q7	1039	609	577	577
   q8	9332	2013	1969	1969
   q9	6921	6320	6316	6316
   q10	8456	3533	3582	3533
   q11	434	223	225	223
   q12	393	200	189	189
   q13	17801	2853	2828	2828
   q14	229	200	207	200
   q15	505	470	457	457
   q16	495	368	365	365
   q17	956	612	524	524
   q18	7099	6453	6409	6409
   q19	3027	1384	1372	1372
   q20	570	249	251	249
   q21	3554	2906	2904	2904
   q22	339	298	287	287
   Total cold run time: 109370 ms
   Total hot run time: 37716 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4108	4082	4051	4051
   q2	335	228	235	228
   q3	2971	2873	2869	2869
   q4	1806	1572	1535	1535
   q5	5185	5229	5234	5229
   q6	196	117	121	117
   q7	2220	1862	1837	1837
   q8	3169	3301	3289	3289
   q9	8465	8424	8418	8418
   q10	3824	3831	3794	3794
   q11	541	439	447	439
   q12	739	568	543	543
   q13	16937	2832	2823	2823
   q14	283	251	256	251
   q15	502	451	450	450
   q16	455	411	409	409
   q17	1735	1477	1440	1440
   q18	7502	7173	7220	7173
   q19	1621	1516	1535	1516
   q20	1905	1739	1702	1702
   q21	4814	4643	4695	4643
   q22	495	426	445	426
   Total cold run time: 69808 ms
   Total hot run time: 53182 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026354187

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542627384


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+    std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+    // workload group id -> workload group queries
+    std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+            all_wg_queries;
+    for (auto& [wg_id, wg] : _workload_groups) {
+        all_wg_queries.insert({wg_id, wg->queries()});
+    }
+
+    int64_t all_queries_mem_used = 0;
+
+    // calculate total memory used of each workload group and total memory used of all queries
+    std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+    for (auto& [wg_id, wg_queries] : all_wg_queries) {
+        int64_t wg_total_mem_used = 0;
+        for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+            if (auto query_ctx = query_ctx_ptr.lock()) {
+                wg_total_mem_used +=
+                        query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();

Review Comment:
   这样可能不行,如果把canceled 设置为0,那么可能会oom,因为此时canceled的query 的内存还没释放。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024719391

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542643876


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+    std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+    // workload group id -> workload group queries
+    std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+            all_wg_queries;
+    for (auto& [wg_id, wg] : _workload_groups) {
+        all_wg_queries.insert({wg_id, wg->queries()});
+    }
+
+    int64_t all_queries_mem_used = 0;
+
+    // calculate total memory used of each workload group and total memory used of all queries
+    std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+    for (auto& [wg_id, wg_queries] : all_wg_queries) {
+        int64_t wg_total_mem_used = 0;
+        for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+            if (auto query_ctx = query_ctx_ptr.lock()) {
+                wg_total_mem_used +=
+                        query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+            }
+        }
+        all_queries_mem_used += wg_total_mem_used;
+        wgs_mem_info[wg_id] = {wg_total_mem_used};
+    }
+
+    auto proc_vm_rss = PerfCounters::get_vm_rss();
+    if (all_queries_mem_used <= 0) {
+        return;
+    }
+
+    auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+    auto sys_mem_available = doris::MemInfo::sys_mem_available();
+    std::string debug_msg = fmt::format(
+            "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+            "{}, all quries mem: {}",
+            PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+            PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+            PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+            PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+    LOG_EVERY_N(INFO, 5) << debug_msg;
+
+    if (proc_vm_rss < all_queries_mem_used) {
+        all_queries_mem_used = proc_vm_rss;
+    }
+
+    // process memory used is actually bigger than all_queries_mem_used,
+    // because memory of page cache, allocator cache, segment cache etc. are included
+    // in process_mem_used.
+    // we count these cache memories equally on workload groups.
+    double ratio = (double)proc_vm_rss / (double)all_queries_mem_used;
+
+    for (auto& wg : _workload_groups) {
+        auto wg_mem_limit = wg.second->memory_limit();
+        auto& wg_mem_info = wgs_mem_info[wg.first];
+        wg_mem_info.weighted_mem_used = wg_mem_info.total_mem_used * ratio;
+        wg_mem_info.mem_used_ratio = (double)wg_mem_info.weighted_mem_used / wg_mem_limit;
+
+        wg.second->set_weighted_memory_used(wg_mem_info.total_mem_used, ratio);
+
+        auto spill_low_water_mark = wg.second->spill_threshold_low_water_mark();
+        auto spill_high_water_mark = wg.second->spill_threashold_high_water_mark();
+        wg_mem_info.is_high_wartermark = (wg_mem_info.weighted_mem_used >
+                                          ((double)wg_mem_limit * spill_high_water_mark / 100));
+        wg_mem_info.is_low_wartermark = (wg_mem_info.weighted_mem_used >
+                                         ((double)wg_mem_limit * spill_low_water_mark / 100));
+
+        // calculate query weighted memory limit of task group
+        const auto& wg_queries = all_wg_queries[wg.first];
+        auto wg_query_count = wg_queries.size();

Review Comment:
   这里如果query count  < 1 感觉可以直接continue了;
   那么216 行直接就是wg_mem_limit ) / wg_query_count



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542650160


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+    std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+    // workload group id -> workload group queries
+    std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+            all_wg_queries;
+    for (auto& [wg_id, wg] : _workload_groups) {
+        all_wg_queries.insert({wg_id, wg->queries()});
+    }
+
+    int64_t all_queries_mem_used = 0;
+
+    // calculate total memory used of each workload group and total memory used of all queries
+    std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+    for (auto& [wg_id, wg_queries] : all_wg_queries) {
+        int64_t wg_total_mem_used = 0;
+        for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+            if (auto query_ctx = query_ctx_ptr.lock()) {
+                wg_total_mem_used +=
+                        query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+            }
+        }
+        all_queries_mem_used += wg_total_mem_used;
+        wgs_mem_info[wg_id] = {wg_total_mem_used};
+    }
+
+    auto proc_vm_rss = PerfCounters::get_vm_rss();
+    if (all_queries_mem_used <= 0) {
+        return;
+    }
+
+    auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+    auto sys_mem_available = doris::MemInfo::sys_mem_available();
+    std::string debug_msg = fmt::format(
+            "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+            "{}, all quries mem: {}",
+            PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+            PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+            PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+            PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+    LOG_EVERY_N(INFO, 5) << debug_msg;

Review Comment:
   这个日志感觉也会刷屏,如果内存不到80% 就别输出了。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026778442

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 182069 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit c8f6c475195c6fa5512777393f6824eb63ce1a44, data reload: false
   
   query1	1199	1112	1096	1096
   query2	6352	2065	1975	1975
   query3	6696	220	218	218
   query4	34666	21377	21436	21377
   query5	4268	375	373	373
   query6	266	182	181	181
   query7	4635	302	291	291
   query8	215	181	177	177
   query9	9790	2212	2202	2202
   query10	447	264	243	243
   query11	17133	14228	14228	14228
   query12	124	83	85	83
   query13	1623	421	416	416
   query14	11057	7042	7040	7040
   query15	207	170	171	170
   query16	6823	250	271	250
   query17	1702	592	555	555
   query18	1390	292	278	278
   query19	244	152	158	152
   query20	90	86	84	84
   query21	199	129	125	125
   query22	4978	4794	4738	4738
   query23	33462	32712	33024	32712
   query24	11309	2882	2959	2882
   query25	646	406	411	406
   query26	1155	157	160	157
   query27	3318	373	390	373
   query28	7946	1878	1942	1878
   query29	916	691	652	652
   query30	305	152	150	150
   query31	1014	756	732	732
   query32	85	57	54	54
   query33	706	263	248	248
   query34	1250	507	516	507
   query35	869	728	720	720
   query36	1034	924	916	916
   query37	110	64	67	64
   query38	3701	3644	3634	3634
   query39	1675	1588	1566	1566
   query40	176	108	112	108
   query41	50	57	45	45
   query42	109	105	100	100
   query43	508	461	471	461
   query44	1344	759	757	757
   query45	280	254	277	254
   query46	1128	734	742	734
   query47	2013	1928	1935	1928
   query48	471	368	368	368
   query49	883	312	344	312
   query50	818	410	411	410
   query51	6885	6768	6892	6768
   query52	106	89	103	89
   query53	354	280	282	280
   query54	279	234	225	225
   query55	88	84	77	77
   query56	238	217	213	213
   query57	1328	1165	1197	1165
   query58	234	201	206	201
   query59	2958	2532	2641	2532
   query60	239	230	223	223
   query61	98	93	104	93
   query62	611	446	442	442
   query63	305	276	275	275
   query64	5684	4076	3984	3984
   query65	3088	3003	2985	2985
   query66	1182	338	328	328
   query67	15545	14738	14806	14738
   query68	9838	527	528	527
   query69	700	374	371	371
   query70	1366	1147	1130	1130
   query71	575	268	270	268
   query72	7051	2565	2361	2361
   query73	1566	326	331	326
   query74	8146	6466	6345	6345
   query75	5191	2246	2266	2246
   query76	6477	903	927	903
   query77	575	228	231	228
   query78	11800	10480	10189	10189
   query79	12092	526	519	519
   query80	1889	353	347	347
   query81	497	210	205	205
   query82	168	82	82	82
   query83	228	137	141	137
   query84	279	77	77	77
   query85	921	325	317	317
   query86	353	286	291	286
   query87	3654	3487	3502	3487
   query88	6185	2314	2306	2306
   query89	483	371	360	360
   query90	2555	171	167	167
   query91	171	153	134	134
   query92	60	43	45	43
   query93	6036	487	482	482
   query94	1616	177	172	172
   query95	408	291	303	291
   query96	628	277	263	263
   query97	2689	2497	2461	2461
   query98	222	213	219	213
   query99	1043	828	854	828
   Total cold run time: 326069 ms
   Total hot run time: 182069 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017713206

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38070 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 0e279b3fac84fecc43a2e6fc9829bd07ba245b4f, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17609	4243	4132	4132
   q2	2119	165	166	165
   q3	10565	1109	1197	1109
   q4	10231	748	799	748
   q5	7458	2965	2995	2965
   q6	204	123	123	123
   q7	1037	613	567	567
   q8	9333	2032	1950	1950
   q9	7469	6644	6598	6598
   q10	8487	3430	3593	3430
   q11	432	227	214	214
   q12	432	204	199	199
   q13	17809	2889	2866	2866
   q14	231	207	213	207
   q15	515	476	460	460
   q16	501	379	372	372
   q17	963	648	538	538
   q18	7175	6607	6545	6545
   q19	1542	1446	1468	1446
   q20	560	253	240	240
   q21	3555	2903	2903	2903
   q22	352	296	293	293
   Total cold run time: 108579 ms
   Total hot run time: 38070 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4107	4089	4099	4089
   q2	342	232	233	232
   q3	2980	2863	2805	2805
   q4	1900	1546	1552	1546
   q5	5343	5354	5336	5336
   q6	195	115	116	115
   q7	2218	1871	1863	1863
   q8	3159	3277	3280	3277
   q9	8715	8729	8694	8694
   q10	3822	3790	3715	3715
   q11	555	461	445	445
   q12	714	530	548	530
   q13	12652	2853	2874	2853
   q14	285	273	257	257
   q15	502	459	464	459
   q16	482	428	421	421
   q17	1735	1508	1481	1481
   q18	7403	7190	7086	7086
   q19	1615	1567	1553	1553
   q20	1899	1754	1717	1717
   q21	4803	4746	4768	4746
   q22	534	454	439	439
   Total cold run time: 65960 ms
   Total hot run time: 53659 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017733857

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 186969 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 0e279b3fac84fecc43a2e6fc9829bd07ba245b4f, data reload: false
   
   query1	936	377	352	352
   query2	7359	2040	1915	1915
   query3	6702	209	214	209
   query4	31799	21232	21659	21232
   query5	4343	419	409	409
   query6	278	175	181	175
   query7	4625	295	301	295
   query8	228	169	174	169
   query9	9298	2251	2265	2251
   query10	576	248	250	248
   query11	17279	14404	14418	14404
   query12	139	85	87	85
   query13	1642	424	425	424
   query14	14907	11366	11591	11366
   query15	317	194	214	194
   query16	8209	263	261	261
   query17	2076	595	558	558
   query18	2108	289	287	287
   query19	368	160	158	158
   query20	94	89	90	89
   query21	210	130	130	130
   query22	4982	4751	4726	4726
   query23	33556	32542	32546	32542
   query24	10721	2898	2897	2897
   query25	632	413	393	393
   query26	1386	157	163	157
   query27	2949	364	364	364
   query28	7430	1914	1885	1885
   query29	892	623	640	623
   query30	306	155	155	155
   query31	954	740	727	727
   query32	95	58	58	58
   query33	741	329	253	253
   query34	910	489	494	489
   query35	828	600	618	600
   query36	1008	876	916	876
   query37	116	78	78	78
   query38	3527	3491	3451	3451
   query39	1486	1402	1420	1402
   query40	224	119	112	112
   query41	48	49	47	47
   query42	103	96	100	96
   query43	470	449	462	449
   query44	1154	737	745	737
   query45	276	272	265	265
   query46	1129	704	700	700
   query47	1917	1861	1862	1861
   query48	461	356	365	356
   query49	1110	361	338	338
   query50	776	375	372	372
   query51	6803	6748	6758	6748
   query52	98	97	90	90
   query53	350	275	271	271
   query54	324	238	245	238
   query55	81	78	80	78
   query56	246	229	230	229
   query57	1204	1123	1136	1123
   query58	226	199	201	199
   query59	2855	2653	2596	2596
   query60	285	244	253	244
   query61	98	107	99	99
   query62	670	458	444	444
   query63	307	278	283	278
   query64	5407	4070	4034	4034
   query65	3103	3016	3036	3016
   query66	1480	376	355	355
   query67	15361	14828	14675	14675
   query68	7045	528	515	515
   query69	617	380	382	380
   query70	1251	1175	1146	1146
   query71	531	288	274	274
   query72	6248	2720	2542	2542
   query73	755	321	311	311
   query74	8543	6640	6736	6640
   query75	4116	2784	2911	2784
   query76	5002	902	884	884
   query77	652	300	270	270
   query78	10776	10286	10080	10080
   query79	6889	521	527	521
   query80	1316	422	419	419
   query81	514	214	221	214
   query82	364	213	208	208
   query83	226	150	152	150
   query84	298	87	85	85
   query85	1142	398	367	367
   query86	373	307	306	306
   query87	3735	3554	3523	3523
   query88	4813	2424	2414	2414
   query89	475	376	364	364
   query90	2088	178	180	178
   query91	172	136	143	136
   query92	65	49	47	47
   query93	5743	493	483	483
   query94	1220	179	176	176
   query95	430	324	332	324
   query96	620	273	280	273
   query97	3059	2901	2872	2872
   query98	232	218	222	218
   query99	1151	930	928	928
   Total cold run time: 313039 ms
   Total hot run time: 186969 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542290286


##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+    if (!need_revoke) {
+        return false;
+    }
+
+    const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+    auto tg = query_ctx->workload_group();
+    bool is_tg_mem_low_water_mark = false;
+    bool is_tg_mem_high_water_mark = false;

Review Comment:
   use wg



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542293320


##########
be/src/runtime/memory/mem_tracker_limiter.h:
##########
@@ -244,6 +245,20 @@ class MemTrackerLimiter final : public MemTracker {
     // Log the memory usage when memory limit is exceeded.
     std::string tracker_limit_exceeded_str();
 
+    void set_weighted_mem(int64_t weighted_limit, double ratio) {

Review Comment:
   这个不要在这里搞。
   这个信息应该是要记录到query context 中的。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026354103

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024907750

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #32641:
URL: https://github.com/apache/doris/pull/32641


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534070171


##########
be/src/vec/exprs/vexpr.cpp:
##########
@@ -97,11 +97,11 @@ TExprNode create_texpr_node_from(const void* data, const PrimitiveType& type, in
         break;
     }
     case TYPE_DATEV2: {
-        THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATEV2>(data, &node));
+        THROW_IF_ERROR(create_texpr_literal_node<TYPE_DATEV2>(data, &node, precision, scale));

Review Comment:
   These code is not related with this PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542283442


##########
be/src/pipeline/exec/partitioned_aggregation_source_operator.cpp:
##########
@@ -128,6 +129,7 @@ Status PartitionedAggSourceOperatorX::close(RuntimeState* state) {
 Status PartitionedAggSourceOperatorX::get_block(RuntimeState* state, vectorized::Block* block,
                                                 bool* eos) {
     auto& local_state = get_local_state(state);
+    local_state.inc_running_big_mem_op_num(state);

Review Comment:
   why do we need check local state's count.
   Only operator is ok?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542541083


##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,47 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    bool need_revoke = query_ctx->need_revoke();
+    if (!need_revoke) {
+        return false;
+    }
+
+    const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+    auto wg = query_ctx->workload_group();
+    if (!wg) {
+        LOG_ONCE(INFO) << "no workload group for query " << print_id(state->query_id());
+        return false;
+    }
+    bool is_wg_mem_low_water_mark = false;
+    bool is_wg_mem_high_water_mark = false;
+    wg->check_mem_used(&is_wg_mem_low_water_mark, &is_wg_mem_high_water_mark);
+    if (is_wg_mem_high_water_mark) {
+        if (revocable_mem_bytes > 0) {
+            LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+            return true;

Review Comment:
   warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
   
   be/src/pipeline/pipeline_x/pipeline_x_task.cpp:324:
   ```diff
   -         if (revocable_mem_bytes > 0) {
   -             LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
   -             return true;
   -         }
   -         return false;
   +         return revocable_mem_bytes > 0;
   ```
   



##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {

Review Comment:
   warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **be/src/runtime/workload_group/workload_group_manager.cpp:146:** 120 lines including whitespace and comments (threshold 80)
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   
   </details>
   



##########
be/src/runtime/query_context.h:
##########
@@ -247,6 +247,32 @@ class QueryContext {
 
     bool is_nereids() const { return _is_nereids; }
 
+    WorkloadGroupPtr workload_group() const { return _workload_group; }
+
+    void inc_running_big_mem_op_num() {
+        _running_big_mem_op_num.fetch_add(1, std::memory_order_relaxed);
+    }
+    void dec_running_big_mem_op_num() {
+        _running_big_mem_op_num.fetch_sub(1, std::memory_order_relaxed);
+    }
+    int32_t get_running_big_mem_op_num() {
+        return _running_big_mem_op_num.load(std::memory_order_relaxed);
+    }
+
+    void set_weighted_mem(int64_t weighted_limit, int64_t weighted_consumption) {
+        std::lock_guard<std::mutex> l(_weighted_mem_lock);
+        _weighted_consumption = weighted_consumption;
+        _weighted_limit = weighted_limit;
+    }
+    void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) {

Review Comment:
   warning: method 'get_weighted_mem_info' can be made const [readability-make-member-function-const]
   
   ```suggestion
       void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) const {
   ```
   



##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -97,7 +121,7 @@
         return _memory_limit > 0;
     }
 
-    Status add_query(TUniqueId query_id) {
+    Status add_query(TUniqueId query_id, std::shared_ptr<QueryContext> query_ctx) {

Review Comment:
   warning: method 'add_query' can be made const [readability-make-member-function-const]
   
   ```suggestion
       Status add_query(TUniqueId query_id, std::shared_ptr<QueryContext> query_ctx) const {
   ```
   



##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +83,29 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
 
     int64_t memory_used();
 
+    int spill_threshold_low_water_mark() const {
+        return _spill_low_watermark.load(std::memory_order_relaxed);
+    }
+    int spill_threashold_high_water_mark() const {
+        return _spill_high_watermark.load(std::memory_order_relaxed);
+    }
+
+    void set_weighted_memory_used(int64_t wg_total_mem_used, double ratio);
+
+    int64_t get_weighted_memory_used() {
+        return _weighted_mem_used.load(std::memory_order_relaxed);
+    }
+
+    void check_mem_used(bool* is_low_wartermark, bool* is_high_wartermark) const {

Review Comment:
   warning: method 'check_mem_used' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
       static void check_mem_used(bool* is_low_wartermark, bool* is_high_wartermark) {
   ```
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542662525


##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,135 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {
+    std::shared_lock<std::shared_mutex> r_lock(_group_mutex);
+    // workload group id -> workload group queries
+    std::unordered_map<uint64_t, std::unordered_map<TUniqueId, std::weak_ptr<QueryContext>>>
+            all_wg_queries;
+    for (auto& [wg_id, wg] : _workload_groups) {
+        all_wg_queries.insert({wg_id, wg->queries()});
+    }
+
+    int64_t all_queries_mem_used = 0;
+
+    // calculate total memory used of each workload group and total memory used of all queries
+    std::unordered_map<uint64_t, WorkloadGroupMemInfo> wgs_mem_info;
+    for (auto& [wg_id, wg_queries] : all_wg_queries) {
+        int64_t wg_total_mem_used = 0;
+        for (const auto& [query_id, query_ctx_ptr] : wg_queries) {
+            if (auto query_ctx = query_ctx_ptr.lock()) {
+                wg_total_mem_used +=
+                        query_ctx->is_cancelled() ? 0 : query_ctx->query_mem_tracker->consumption();
+            }
+        }
+        all_queries_mem_used += wg_total_mem_used;
+        wgs_mem_info[wg_id] = {wg_total_mem_used};
+    }
+
+    auto proc_vm_rss = PerfCounters::get_vm_rss();
+    if (all_queries_mem_used <= 0) {
+        return;
+    }
+
+    auto process_mem_used = doris::MemInfo::proc_mem_no_allocator_cache();
+    auto sys_mem_available = doris::MemInfo::sys_mem_available();
+    std::string debug_msg = fmt::format(
+            "\nProcess Memory Summary: process_vm_rss: {}, process mem: {}, sys mem available: "
+            "{}, all quries mem: {}",
+            PrettyPrinter::print(proc_vm_rss, TUnit::BYTES),
+            PrettyPrinter::print(process_mem_used, TUnit::BYTES),
+            PrettyPrinter::print(sys_mem_available, TUnit::BYTES),
+            PrettyPrinter::print(all_queries_mem_used, TUnit::BYTES));
+    LOG_EVERY_N(INFO, 5) << debug_msg;
+
+    if (proc_vm_rss < all_queries_mem_used) {
+        all_queries_mem_used = proc_vm_rss;
+    }
+
+    // process memory used is actually bigger than all_queries_mem_used,
+    // because memory of page cache, allocator cache, segment cache etc. are included
+    // in process_mem_used.
+    // we count these cache memories equally on workload groups.
+    double ratio = (double)proc_vm_rss / (double)all_queries_mem_used;
+
+    for (auto& wg : _workload_groups) {
+        auto wg_mem_limit = wg.second->memory_limit();
+        auto& wg_mem_info = wgs_mem_info[wg.first];
+        wg_mem_info.weighted_mem_used = wg_mem_info.total_mem_used * ratio;
+        wg_mem_info.mem_used_ratio = (double)wg_mem_info.weighted_mem_used / wg_mem_limit;
+
+        wg.second->set_weighted_memory_used(wg_mem_info.total_mem_used, ratio);
+
+        auto spill_low_water_mark = wg.second->spill_threshold_low_water_mark();
+        auto spill_high_water_mark = wg.second->spill_threashold_high_water_mark();
+        wg_mem_info.is_high_wartermark = (wg_mem_info.weighted_mem_used >
+                                          ((double)wg_mem_limit * spill_high_water_mark / 100));
+        wg_mem_info.is_low_wartermark = (wg_mem_info.weighted_mem_used >
+                                         ((double)wg_mem_limit * spill_low_water_mark / 100));
+
+        // calculate query weighted memory limit of task group
+        const auto& wg_queries = all_wg_queries[wg.first];
+        auto wg_query_count = wg_queries.size();
+        int64_t query_weighted_mem_limit =
+                wg_query_count ? (wg_mem_limit + wg_query_count) / wg_query_count : wg_mem_limit;
+
+        debug_msg = "";
+        if (wg_mem_info.is_high_wartermark || wg_mem_info.is_low_wartermark) {
+            debug_msg = fmt::format(
+                    "\nWorkload Group {}: mem limit: {}, mem used: {}, weighted mem used: {}, used "
+                    "ratio: {}, query "
+                    "count: {}, query_weighted_mem_limit: {}",
+                    wg.second->name(), PrettyPrinter::print(wg_mem_limit, TUnit::BYTES),
+                    PrettyPrinter::print(wg_mem_info.total_mem_used, TUnit::BYTES),
+                    PrettyPrinter::print(wg_mem_info.weighted_mem_used, TUnit::BYTES),
+                    wg_mem_info.mem_used_ratio, wg_query_count,
+                    PrettyPrinter::print(query_weighted_mem_limit, TUnit::BYTES));
+
+            debug_msg += "\n  Query Memory Summary:";
+        }
+        // check where queries need to revoke memory for task group
+        for (const auto& query : wg_queries) {
+            auto query_ctx = query.second.lock();
+            if (!query_ctx) {
+                continue;
+            }
+            auto query_consumption = query_ctx->query_mem_tracker->consumption();
+            int64_t query_weighted_consumption = query_consumption * ratio;
+            query_ctx->set_weighted_mem(query_weighted_mem_limit, query_weighted_consumption);
+
+            bool need_revoke = false;
+            if (wg_mem_info.is_high_wartermark) {
+                need_revoke = true;
+            } else if (wg_mem_info.is_low_wartermark) {
+                need_revoke = query_weighted_consumption > query_weighted_mem_limit;
+            }
+            query_ctx->set_need_revoke(need_revoke);
+
+            if (wg_mem_info.is_high_wartermark || wg_mem_info.is_low_wartermark) {
+                debug_msg += fmt::format(
+                        "\n    MemTracker Label={}, Parent Label={}, Used={}, WeightedUsed={}, "
+                        "Peak={}",
+                        query_ctx->query_mem_tracker->label(),
+                        query_ctx->query_mem_tracker->parent_label(),
+                        PrettyPrinter::print(query_consumption, TUnit::BYTES),
+                        PrettyPrinter::print(query_weighted_consumption, TUnit::BYTES),
+                        PrettyPrinter::print(query_ctx->query_mem_tracker->peak_consumption(),
+                                             TUnit::BYTES));
+            }
+        }
+        if (wg_mem_info.is_high_wartermark || wg_mem_info.is_low_wartermark) {
+            LOG_EVERY_N(INFO, 3) << debug_msg;

Review Comment:
   改成10 吧



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542293516


##########
be/src/runtime/memory/mem_tracker_limiter.h:
##########
@@ -244,6 +245,20 @@ class MemTrackerLimiter final : public MemTracker {
     // Log the memory usage when memory limit is exceeded.
     std::string tracker_limit_exceeded_str();
 
+    void set_weighted_mem(int64_t weighted_limit, double ratio) {
+        std::lock_guard<std::mutex> l(_weighted_mem_lock);
+        _weighted_consumption = _consumption->current_value() * ratio;
+        _weighted_limit = weighted_limit;
+    }
+    void get_weighted_mem_info(int64_t& weighted_limit, int64_t& weighted_consumption) {
+        std::lock_guard<std::mutex> l(_weighted_mem_lock);
+        weighted_limit = _weighted_limit;
+        weighted_consumption = _weighted_consumption;
+    }
+
+    void set_need_revoke(bool b) { _need_revoke.store(b, std::memory_order_relaxed); }

Review Comment:
   这些函数应该都是要再query ctx 中



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2026869495

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534078055


##########
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##########
@@ -462,6 +472,10 @@ Status PartitionedHashJoinProbeOperatorX::open(RuntimeState* state) {
 Status PartitionedHashJoinProbeOperatorX::push(RuntimeState* state, vectorized::Block* input_block,
                                                bool eos) const {
     auto& local_state = get_local_state(state);
+    if (!local_state._is_running) {

Review Comment:
   You could implement 2 method in parent class:
   inc_running_big_mem_op_num(){
   if (not add) {
   get_query_ctx()->inc_running_big_mem_op_num;
   not_add=true;
   }
   }
   
   dec_big_mem_op_num(){
   if (not_add) {
   log_fatal<<} else if (!_dec) {
   query_ctx->dec;
   _dec=true;
   }
   }



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017575653

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534078055


##########
be/src/pipeline/exec/partitioned_hash_join_probe_operator.cpp:
##########
@@ -462,6 +472,10 @@ Status PartitionedHashJoinProbeOperatorX::open(RuntimeState* state) {
 Status PartitionedHashJoinProbeOperatorX::push(RuntimeState* state, vectorized::Block* input_block,
                                                bool eos) const {
     auto& local_state = get_local_state(state);
+    if (!local_state._is_running) {

Review Comment:
   You could implement 2 method in parent class:
   inc_running_big_mem_op_num(){
   if (not add) {
   get_query_ctx()->inc_running_big_mem_op_num;
   not_add=true;
   }
   }
   
   dec_big_mem_op_num(){
   if (not_add) {
   log_fatal<<} else if (!_dec) {
   query_ctx->dec;
   _dec=true;
   }
   }
   
   then all subclass could use this method.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021926523

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 38470 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit e1783354448196d335626f926218759a3286ec2e, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	18359	4551	4252	4252
   q2	2595	166	177	166
   q3	11115	1158	1219	1158
   q4	10864	825	849	825
   q5	7774	3081	3066	3066
   q6	218	129	128	128
   q7	1078	643	588	588
   q8	9590	2073	2042	2042
   q9	7348	6672	6573	6573
   q10	8380	3424	3574	3424
   q11	418	233	217	217
   q12	369	211	203	203
   q13	17795	2904	2870	2870
   q14	241	203	200	200
   q15	505	460	454	454
   q16	470	381	373	373
   q17	967	529	626	529
   q18	7174	6528	6574	6528
   q19	1560	1434	1492	1434
   q20	545	249	247	247
   q21	3648	2978	2909	2909
   q22	356	284	292	284
   Total cold run time: 111369 ms
   Total hot run time: 38470 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4114	4084	4107	4084
   q2	328	227	231	227
   q3	2983	2860	2961	2860
   q4	1902	1593	1600	1593
   q5	5331	5346	5356	5346
   q6	194	115	117	115
   q7	2288	1856	1885	1856
   q8	3175	3292	3331	3292
   q9	8810	8731	8788	8731
   q10	3830	3793	3798	3793
   q11	543	446	449	446
   q12	714	544	541	541
   q13	16567	2865	2880	2865
   q14	279	248	270	248
   q15	499	459	462	459
   q16	480	429	430	429
   q17	1747	1526	1496	1496
   q18	7528	7201	7244	7201
   q19	1623	1548	1511	1511
   q20	1891	1737	1702	1702
   q21	4893	4804	4733	4733
   q22	548	430	427	427
   Total cold run time: 70267 ms
   Total hot run time: 53955 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021945104

   TeamCity be ut coverage result:
    Function Coverage: 35.22% (8741/24816) 
    Line Coverage: 27.01% (71551/264859)
    Region Coverage: 26.27% (37132/141340)
    Branch Coverage: 23.16% (18982/81952)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/e1783354448196d335626f926218759a3286ec2e_e1783354448196d335626f926218759a3286ec2e/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2021945803

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit e1783354448196d335626f926218759a3286ec2e with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542290653


##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+    if (!need_revoke) {
+        return false;
+    }
+
+    const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+    auto tg = query_ctx->workload_group();
+    bool is_tg_mem_low_water_mark = false;
+    bool is_tg_mem_high_water_mark = false;
+    if (tg) {
+        tg->check_mem_used(is_tg_mem_low_water_mark, is_tg_mem_high_water_mark);
+    }
+    if (is_tg_mem_high_water_mark) {
+        if (revocable_mem_bytes > 1024 * 1024) {

Review Comment:
   change this to an session variable.
   not use magic numbers.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024274053

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024884658

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 180802 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 09207c21e85cb9bb7da144b2960060fec2a8e348, data reload: false
   
   query1	947	363	353	353
   query2	6555	1840	1919	1840
   query3	6704	210	207	207
   query4	31834	21276	21273	21273
   query5	4319	413	385	385
   query6	273	177	172	172
   query7	4624	284	293	284
   query8	226	167	173	167
   query9	9413	2351	2349	2349
   query10	572	246	255	246
   query11	15396	14080	14224	14080
   query12	135	97	87	87
   query13	1641	426	411	411
   query14	9458	7670	7935	7670
   query15	261	187	185	185
   query16	8232	265	269	265
   query17	1958	552	527	527
   query18	2095	274	282	274
   query19	353	147	150	147
   query20	91	85	88	85
   query21	209	126	121	121
   query22	4992	4829	4775	4775
   query23	33739	32940	32803	32803
   query24	11018	2811	2888	2811
   query25	588	360	367	360
   query26	1189	146	148	146
   query27	2978	344	349	344
   query28	7646	1948	1921	1921
   query29	862	632	605	605
   query30	300	151	162	151
   query31	999	704	736	704
   query32	90	58	51	51
   query33	747	251	232	232
   query34	1043	482	499	482
   query35	820	612	599	599
   query36	1014	888	877	877
   query37	127	63	61	61
   query38	3528	3389	3417	3389
   query39	1468	1461	1432	1432
   query40	218	109	110	109
   query41	48	47	45	45
   query42	100	98	93	93
   query43	489	438	439	438
   query44	1203	710	715	710
   query45	282	255	266	255
   query46	1094	704	695	695
   query47	1899	1821	1840	1821
   query48	447	366	358	358
   query49	1094	325	323	323
   query50	766	366	372	366
   query51	6651	6533	6549	6533
   query52	105	96	93	93
   query53	349	270	274	270
   query54	286	221	226	221
   query55	94	78	76	76
   query56	247	213	214	213
   query57	1211	1129	1119	1119
   query58	226	196	203	196
   query59	2638	2504	2472	2472
   query60	266	228	264	228
   query61	111	108	113	108
   query62	643	404	434	404
   query63	310	282	279	279
   query64	5715	4018	4077	4018
   query65	3137	3031	2998	2998
   query66	1451	377	378	377
   query67	15188	14777	14781	14777
   query68	6800	532	533	532
   query69	609	395	371	371
   query70	1183	1184	1115	1115
   query71	462	264	265	264
   query72	6499	2698	2463	2463
   query73	732	324	323	323
   query74	7751	6429	6407	6407
   query75	3456	2207	2219	2207
   query76	4002	906	888	888
   query77	602	269	256	256
   query78	10836	10140	10218	10140
   query79	8798	519	519	519
   query80	1555	391	358	358
   query81	532	218	219	218
   query82	864	84	85	84
   query83	219	143	139	139
   query84	289	82	81	81
   query85	1502	316	308	308
   query86	468	320	285	285
   query87	3741	3508	3513	3508
   query88	5106	2378	2369	2369
   query89	495	370	364	364
   query90	1937	180	179	179
   query91	173	137	141	137
   query92	61	47	46	46
   query93	6948	493	482	482
   query94	1154	175	175	175
   query95	430	321	326	321
   query96	599	274	275	274
   query97	2630	2477	2466	2466
   query98	237	214	212	212
   query99	1257	845	855	845
   Total cold run time: 306639 ms
   Total hot run time: 180802 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024992428

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 180902 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit df0ddde4d7f8ba7db99389a85bb281abe6306fe4, data reload: false
   
   query1	947	367	361	361
   query2	6553	1960	1937	1937
   query3	6705	208	204	204
   query4	31963	21304	21329	21304
   query5	4318	399	402	399
   query6	270	180	173	173
   query7	4648	293	290	290
   query8	229	170	168	168
   query9	9364	2319	2330	2319
   query10	560	253	262	253
   query11	17206	14160	14161	14160
   query12	138	95	93	93
   query13	1626	421	420	420
   query14	9248	7587	7144	7144
   query15	219	186	179	179
   query16	8033	255	259	255
   query17	1959	557	536	536
   query18	2051	294	281	281
   query19	235	148	146	146
   query20	91	90	85	85
   query21	200	123	123	123
   query22	4979	4785	4768	4768
   query23	33557	33171	32684	32684
   query24	12097	2899	2845	2845
   query25	665	400	390	390
   query26	1841	153	153	153
   query27	3184	320	314	314
   query28	8179	1917	1900	1900
   query29	1056	649	614	614
   query30	303	152	147	147
   query31	987	730	743	730
   query32	95	58	58	58
   query33	763	267	262	262
   query34	1114	487	507	487
   query35	920	700	683	683
   query36	1044	881	915	881
   query37	268	69	65	65
   query38	3553	3443	3433	3433
   query39	1575	1571	1515	1515
   query40	283	106	112	106
   query41	52	50	45	45
   query42	103	92	98	92
   query43	494	454	436	436
   query44	1262	708	716	708
   query45	268	266	239	239
   query46	1096	706	687	687
   query47	1944	1839	1839	1839
   query48	451	359	358	358
   query49	1232	333	345	333
   query50	767	374	374	374
   query51	6693	6638	6560	6560
   query52	120	93	99	93
   query53	341	274	276	274
   query54	318	237	233	233
   query55	84	83	80	80
   query56	262	238	235	235
   query57	1238	1116	1122	1116
   query58	230	207	212	207
   query59	2880	2565	2561	2561
   query60	264	250	263	250
   query61	123	114	114	114
   query62	692	436	438	436
   query63	297	272	275	272
   query64	6908	4119	4107	4107
   query65	3079	3017	3025	3017
   query66	1417	379	369	369
   query67	15660	14902	14783	14783
   query68	9214	534	541	534
   query69	708	376	377	376
   query70	1420	1142	1194	1142
   query71	524	267	261	261
   query72	6525	2530	2380	2380
   query73	1625	330	324	324
   query74	8143	6467	6438	6438
   query75	3895	2271	2206	2206
   query76	5682	864	848	848
   query77	646	251	254	251
   query78	11020	10259	10058	10058
   query79	10681	516	517	516
   query80	1859	371	375	371
   query81	509	213	227	213
   query82	322	85	85	85
   query83	216	144	143	143
   query84	291	77	79	77
   query85	1074	308	297	297
   query86	356	305	300	300
   query87	3742	3626	3536	3536
   query88	4760	2349	2345	2345
   query89	486	358	372	358
   query90	2007	175	171	171
   query91	178	141	137	137
   query92	64	48	47	47
   query93	6890	508	487	487
   query94	1332	177	177	177
   query95	424	326	321	321
   query96	640	264	266	264
   query97	2633	2515	2493	2493
   query98	236	222	213	213
   query99	1121	861	850	850
   Total cold run time: 320524 ms
   Total hot run time: 180902 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2017739697

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 0e279b3fac84fecc43a2e6fc9829bd07ba245b4f with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
   Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1537294093


##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -82,6 +82,31 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
 
     int64_t memory_used();
 
+    int spill_threshold_low_water_mark() const {
+        return _spill_low_watermark.load(std::memory_order_relaxed);
+    }
+    int spill_threashold_high_water_mark() const {
+        return _spill_high_watermark.load(std::memory_order_relaxed);
+    }
+
+    void set_weighted_memory_used(int64_t wg_total_mem_used, double ratio);
+
+    std::unordered_set<std::shared_ptr<MemTrackerLimiter>> get_all_mem_trackers();
+
+    int64_t get_weighted_memory_used() {
+        return _weighted_mem_used.load(std::memory_order_relaxed);
+    }
+
+    void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {

Review Comment:
   warning: method 'check_mem_used' can be made static [readability-convert-member-functions-to-static]
   
   ```suggestion
       static void check_mem_used(bool& is_low_wartermark, bool& is_high_wartermark) {
   ```
   



##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@ void WorkloadGroupMgr::delete_workload_group_by_ids(std::set<uint64_t> used_wg_i
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {

Review Comment:
   warning: method 'refresh_wg_memory_info' can be made static [readability-convert-member-functions-to-static]
   
   be/src/runtime/workload_group/workload_group_manager.h:56:
   ```diff
   -     void refresh_wg_memory_info();
   +     static void refresh_wg_memory_info();
   ```
   



##########
be/src/pipeline/pipeline_x/pipeline_x_task.cpp:
##########
@@ -326,6 +304,46 @@ Status PipelineXTask::execute(bool* eos) {
     return status;
 }
 
+bool PipelineXTask::should_revoke_memory(RuntimeState* state, int64_t revocable_mem_bytes) {
+    auto* query_ctx = state->get_query_ctx();
+    bool need_revoke = query_ctx->query_mem_tracker->need_revoke();
+    if (!need_revoke) {
+        return false;
+    }
+
+    const auto min_revocable_mem_bytes = state->min_revocable_mem();
+
+    auto tg = query_ctx->workload_group();
+    bool is_tg_mem_low_water_mark = false;
+    bool is_tg_mem_high_water_mark = false;
+    if (tg) {
+        tg->check_mem_used(is_tg_mem_low_water_mark, is_tg_mem_high_water_mark);
+    }
+    if (is_tg_mem_high_water_mark) {
+        if (revocable_mem_bytes > 1024 * 1024) {
+            LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
+            return true;

Review Comment:
   warning: redundant boolean literal in conditional return statement [readability-simplify-boolean-expr]
   
   be/src/pipeline/pipeline_x/pipeline_x_task.cpp:322:
   ```diff
   -         if (revocable_mem_bytes > 1024 * 1024) {
   -             LOG_EVERY_N(INFO, 5) << "revoke memory, hight water mark";
   -             return true;
   -         }
   -         return false;
   +         return revocable_mem_bytes > 1024 * 1024;
   ```
   



##########
be/src/runtime/workload_group/workload_group_manager.cpp:
##########
@@ -135,6 +137,121 @@
               << "ms, deleted group size:" << deleted_task_groups.size();
 }
 
+struct WorkloadGroupMemInfo {
+    int64_t total_mem_used = 0;
+    int64_t weighted_mem_used = 0;
+    bool is_low_wartermark = false;
+    bool is_high_wartermark = false;
+    double mem_used_ratio = 0;
+};
+void WorkloadGroupMgr::refresh_wg_memory_info() {

Review Comment:
   warning: function 'refresh_wg_memory_info' exceeds recommended size/complexity thresholds [readability-function-size]
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   <details>
   <summary>Additional context</summary>
   
   **be/src/runtime/workload_group/workload_group_manager.cpp:146:** 106 lines including whitespace and comments (threshold 80)
   ```cpp
   void WorkloadGroupMgr::refresh_wg_memory_info() {
                          ^
   ```
   
   </details>
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1534069010


##########
be/src/vec/spill/spill_writer.cpp:
##########
@@ -35,13 +35,19 @@ Status SpillWriter::open() {
     return Status::OK();
 }
 
+SpillWriter::~SpillWriter() {
+    if (!closed_) {
+        LOG(WARNING) << "spill writer not closed correctly:\n" << boost::stacktrace::stacktrace();

Review Comment:
   not use this way, boost has a static global lock.
   Using doris::Status wrapped error stack, you could use it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2025041379

   TeamCity be ut coverage result:
    Function Coverage: 35.59% (8837/24827) 
    Line Coverage: 27.30% (72396/265177)
    Region Coverage: 26.54% (37555/141484)
    Branch Coverage: 23.35% (19151/82034)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/df0ddde4d7f8ba7db99389a85bb281abe6306fe4_df0ddde4d7f8ba7db99389a85bb281abe6306fe4/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024336558

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 615fff13fc8d598fd9aad875a0fb730eabc282d0 with default session variables
   Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #32641:
URL: https://github.com/apache/doris/pull/32641#discussion_r1542352340


##########
be/src/runtime/workload_group/workload_group.h:
##########
@@ -141,7 +166,8 @@ class WorkloadGroup : public std::enable_shared_from_this<WorkloadGroup> {
     const uint64_t _id;
     std::string _name;
     int64_t _version;
-    int64_t _memory_limit; // bytes
+    int64_t _memory_limit;                      // bytes
+    std::atomic_int64_t _weighted_mem_used = 0; // bytes
     bool _enable_memory_overcommit;
     std::atomic<uint64_t> _cpu_share;

Review Comment:
   remove query_id_set and use std::map<queryid, weak_ptr<query context>> so that all memory limit info could be set to query context, not to memtracker.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [improvement](spill) spill trigger improvement [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #32641:
URL: https://github.com/apache/doris/pull/32641#issuecomment-2024897391

   
   Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   ```
   Load test result on commit 09207c21e85cb9bb7da144b2960060fec2a8e348 with default session variables
   Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
   Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
   Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
   Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org