You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2024/06/13 05:43:35 UTC

[PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

xinyiZzz opened a new pull request, #36235:
URL: https://github.com/apache/doris/pull/36235

   ## Proposed changes
   
   Fix #35590, CacheManager get CachePolicy should be used lock.
   - Add LRUCachePolicyTrackingAllocator and LRUCachePolicyTrackingManual.
   - tracking memory in LRUCachePolicy::insert, LRUCacheValueBase not need init `mem_tracker`, so not need to get CachePolicy under lock.
   
   <!--Describe your changes.-->
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #36235:
URL: https://github.com/apache/doris/pull/36235#discussion_r1644643785


##########
be/src/runtime/memory/lru_cache_policy.h:
##########
@@ -208,6 +195,116 @@ class LRUCachePolicy : public CachePolicy {
         }
     }
 
+protected:
+    // if check_capacity failed, will return dummy lru cache,
+    // compatible with ShardedLRUCache usage, but will not actually cache.
+    std::shared_ptr<Cache> _cache;
+    LRUCacheType _lru_cache_type;
+};
+
+class LRUCachePolicyTrackingAllocator : public LRUCachePolicy {
+public:
+    LRUCachePolicyTrackingAllocator(
+            CacheType type, size_t capacity, LRUCacheType lru_cache_type,
+            uint32_t stale_sweep_time_s, uint32_t num_shards = DEFAULT_LRU_CACHE_NUM_SHARDS,
+            uint32_t element_count_capacity = DEFAULT_LRU_CACHE_ELEMENT_COUNT_CAPACITY,
+            bool enable_prune = true)
+            : LRUCachePolicy(type, capacity, lru_cache_type, stale_sweep_time_s, num_shards,
+                             element_count_capacity, enable_prune) {
+        init_mem_tracker(lru_cache_type_string(lru_cache_type));
+    }
+
+    LRUCachePolicyTrackingAllocator(CacheType type, size_t capacity, LRUCacheType lru_cache_type,
+                                    uint32_t stale_sweep_time_s, uint32_t num_shards,
+                                    uint32_t element_count_capacity,
+                                    CacheValueTimeExtractor cache_value_time_extractor,
+                                    bool cache_value_check_timestamp, bool enable_prune = true)
+            : LRUCachePolicy(type, capacity, lru_cache_type, stale_sweep_time_s, num_shards,
+                             element_count_capacity, cache_value_time_extractor,
+                             cache_value_check_timestamp, enable_prune) {
+        init_mem_tracker(lru_cache_type_string(lru_cache_type));
+    }
+
+    ~LRUCachePolicyTrackingAllocator() override { reset_cache(); }
+
+    void init_mem_tracker(const std::string& type_name) {

Review Comment:
   should be private `_init_mem_tracker`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167565397

   
   <details>
   <summary>TPC-DS: <b>Total hot run time: 171870 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 1bb4f14c1c87bb8972bf9565ec19c30323d92e2b, data reload: false
   
   query1	936	384	379	379
   query2	6474	2378	2228	2228
   query3	6651	205	210	205
   query4	19088	17523	17574	17523
   query5	4142	475	460	460
   query6	262	162	160	160
   query7	4601	291	288	288
   query8	315	301	307	301
   query9	8580	2395	2375	2375
   query10	591	287	275	275
   query11	10657	10152	9937	9937
   query12	132	89	85	85
   query13	1639	355	361	355
   query14	9067	6130	6304	6130
   query15	231	198	191	191
   query16	7328	272	272	272
   query17	1401	578	527	527
   query18	1950	278	283	278
   query19	199	158	158	158
   query20	89	86	85	85
   query21	208	130	127	127
   query22	4419	4394	3996	3996
   query23	33667	32905	33095	32905
   query24	12446	2813	2894	2813
   query25	696	355	356	355
   query26	1903	152	155	152
   query27	3213	320	311	311
   query28	7835	2031	2021	2021
   query29	1330	626	615	615
   query30	283	151	152	151
   query31	950	729	771	729
   query32	92	53	55	53
   query33	787	295	275	275
   query34	1009	476	456	456
   query35	731	614	632	614
   query36	1106	934	963	934
   query37	283	68	70	68
   query38	2911	2758	2788	2758
   query39	858	802	796	796
   query40	283	126	126	126
   query41	55	52	56	52
   query42	119	101	105	101
   query43	575	521	526	521
   query44	1214	730	742	730
   query45	195	159	166	159
   query46	1085	713	715	713
   query47	1878	1798	1773	1773
   query48	386	327	295	295
   query49	1202	419	404	404
   query50	769	372	382	372
   query51	6942	6775	6700	6700
   query52	99	91	95	91
   query53	354	282	284	282
   query54	949	434	433	433
   query55	77	80	76	76
   query56	280	254	268	254
   query57	1159	1039	1039	1039
   query58	252	273	241	241
   query59	3489	3179	3008	3008
   query60	298	272	280	272
   query61	100	94	95	94
   query62	658	447	460	447
   query63	312	294	284	284
   query64	9999	2275	1748	1748
   query65	3174	3087	3113	3087
   query66	1419	336	328	328
   query67	15397	14981	14955	14955
   query68	4528	533	539	533
   query69	460	316	313	313
   query70	1194	1142	1127	1127
   query71	400	278	267	267
   query72	7254	5410	5512	5410
   query73	755	323	319	319
   query74	5990	5590	5542	5542
   query75	3408	2698	2689	2689
   query76	2687	968	972	968
   query77	468	309	304	304
   query78	10694	9867	9833	9833
   query79	2517	508	512	508
   query80	1096	466	462	462
   query81	579	227	224	224
   query82	758	105	97	97
   query83	238	169	168	168
   query84	238	85	85	85
   query85	1632	291	285	285
   query86	497	317	306	306
   query87	3274	3086	3090	3086
   query88	4237	2362	2363	2362
   query89	494	376	388	376
   query90	1775	198	184	184
   query91	126	100	101	100
   query92	69	48	49	48
   query93	1795	506	496	496
   query94	1265	199	186	186
   query95	404	300	314	300
   query96	586	261	262	261
   query97	3208	3063	3068	3063
   query98	228	206	198	198
   query99	1209	864	844	844
   Total cold run time: 275842 ms
   Total hot run time: 171870 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2185620264

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz merged PR #36235:
URL: https://github.com/apache/doris/pull/36235


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167447786

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2164481302

   clang-tidy review says "All clean, LGTM! :+1:"


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167574678

   
   <details>
   <summary>ClickBench: <b>Total hot run time: 30.72 s</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 1bb4f14c1c87bb8972bf9565ec19c30323d92e2b, data reload: false
   
   query1	0.05	0.03	0.03
   query2	0.08	0.03	0.04
   query3	0.23	0.06	0.05
   query4	1.67	0.08	0.10
   query5	0.50	0.50	0.49
   query6	1.11	0.72	0.73
   query7	0.02	0.02	0.01
   query8	0.05	0.05	0.04
   query9	0.54	0.50	0.48
   query10	0.54	0.55	0.55
   query11	0.15	0.12	0.11
   query12	0.16	0.13	0.12
   query13	0.59	0.58	0.59
   query14	0.81	0.79	0.77
   query15	0.84	0.81	0.82
   query16	0.38	0.36	0.36
   query17	0.95	0.97	0.96
   query18	0.23	0.23	0.26
   query19	1.80	1.84	1.66
   query20	0.01	0.01	0.01
   query21	15.44	0.63	0.64
   query22	5.08	5.94	2.37
   query23	18.26	1.29	1.17
   query24	2.09	0.21	0.21
   query25	0.15	0.08	0.08
   query26	0.26	0.17	0.18
   query27	0.08	0.09	0.07
   query28	13.27	1.04	0.99
   query29	12.62	3.27	3.23
   query30	0.26	0.06	0.06
   query31	2.88	0.39	0.38
   query32	3.26	0.47	0.47
   query33	2.86	2.96	2.89
   query34	17.18	4.42	4.50
   query35	4.47	4.46	4.44
   query36	0.65	0.47	0.46
   query37	0.18	0.15	0.15
   query38	0.16	0.14	0.14
   query39	0.04	0.03	0.03
   query40	0.18	0.15	0.15
   query41	0.09	0.04	0.05
   query42	0.07	0.05	0.04
   query43	0.04	0.04	0.04
   Total cold run time: 110.28 s
   Total hot run time: 30.72 s
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167435997

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2185620296

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2164457648

   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
   
   Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
   See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2164460176

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167566527

   TeamCity be ut coverage result:
    Function Coverage: 36.44% (8985/24654) 
    Line Coverage: 28.03% (73679/262902)
    Region Coverage: 27.50% (38276/139182)
    Branch Coverage: 24.20% (19521/80650)
    Coverage Report: http://coverage.selectdb-in.cc/coverage/1bb4f14c1c87bb8972bf9565ec19c30323d92e2b_1bb4f14c1c87bb8972bf9565ec19c30323d92e2b/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]

Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167546132

   
   <details>
   <summary>TPC-H: <b>Total hot run time: 39817 ms</b></summary>
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 1bb4f14c1c87bb8972bf9565ec19c30323d92e2b, data reload: false
   
   ------ Round 1 ----------------------------------
   q1	17618	4491	4317	4317
   q2	2035	195	192	192
   q3	10466	1122	1229	1122
   q4	10186	860	693	693
   q5	7475	2695	2639	2639
   q6	221	137	134	134
   q7	963	607	591	591
   q8	9219	2094	2139	2094
   q9	9203	6529	6531	6529
   q10	8985	3738	3692	3692
   q11	459	236	240	236
   q12	519	225	227	225
   q13	18930	2972	2939	2939
   q14	270	229	218	218
   q15	520	495	494	494
   q16	516	377	387	377
   q17	998	654	667	654
   q18	8081	7424	7355	7355
   q19	1699	1406	1485	1406
   q20	642	309	319	309
   q21	5039	3265	3946	3265
   q22	383	336	337	336
   Total cold run time: 114427 ms
   Total hot run time: 39817 ms
   
   ----- Round 2, with runtime_filter_mode=off -----
   q1	4302	4283	4257	4257
   q2	367	262	259	259
   q3	2968	2824	2742	2742
   q4	1895	1604	1617	1604
   q5	5258	5295	5287	5287
   q6	210	126	127	126
   q7	2122	1759	1777	1759
   q8	3219	3369	3348	3348
   q9	8446	8436	8419	8419
   q10	3907	3623	3696	3623
   q11	591	508	500	500
   q12	786	608	595	595
   q13	17447	2968	2997	2968
   q14	294	278	252	252
   q15	537	474	484	474
   q16	462	400	409	400
   q17	1774	1506	1490	1490
   q18	7566	7625	7387	7387
   q19	1717	1674	1585	1585
   q20	1997	1787	1776	1776
   q21	4830	4795	4726	4726
   q22	613	580	534	534
   Total cold run time: 71308 ms
   Total hot run time: 54111 ms
   ```
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org