You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "xinyiZzz (via GitHub)" <gi...@apache.org> on 2024/06/13 05:43:35 UTC
[PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
xinyiZzz opened a new pull request, #36235:
URL: https://github.com/apache/doris/pull/36235
## Proposed changes
Fix #35590, CacheManager get CachePolicy should be used lock.
- Add LRUCachePolicyTrackingAllocator and LRUCachePolicyTrackingManual.
- tracking memory in LRUCachePolicy::insert, LRUCacheValueBase not need init `mem_tracker`, so not need to get CachePolicy under lock.
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #36235:
URL: https://github.com/apache/doris/pull/36235#discussion_r1644643785
##########
be/src/runtime/memory/lru_cache_policy.h:
##########
@@ -208,6 +195,116 @@ class LRUCachePolicy : public CachePolicy {
}
}
+protected:
+ // if check_capacity failed, will return dummy lru cache,
+ // compatible with ShardedLRUCache usage, but will not actually cache.
+ std::shared_ptr<Cache> _cache;
+ LRUCacheType _lru_cache_type;
+};
+
+class LRUCachePolicyTrackingAllocator : public LRUCachePolicy {
+public:
+ LRUCachePolicyTrackingAllocator(
+ CacheType type, size_t capacity, LRUCacheType lru_cache_type,
+ uint32_t stale_sweep_time_s, uint32_t num_shards = DEFAULT_LRU_CACHE_NUM_SHARDS,
+ uint32_t element_count_capacity = DEFAULT_LRU_CACHE_ELEMENT_COUNT_CAPACITY,
+ bool enable_prune = true)
+ : LRUCachePolicy(type, capacity, lru_cache_type, stale_sweep_time_s, num_shards,
+ element_count_capacity, enable_prune) {
+ init_mem_tracker(lru_cache_type_string(lru_cache_type));
+ }
+
+ LRUCachePolicyTrackingAllocator(CacheType type, size_t capacity, LRUCacheType lru_cache_type,
+ uint32_t stale_sweep_time_s, uint32_t num_shards,
+ uint32_t element_count_capacity,
+ CacheValueTimeExtractor cache_value_time_extractor,
+ bool cache_value_check_timestamp, bool enable_prune = true)
+ : LRUCachePolicy(type, capacity, lru_cache_type, stale_sweep_time_s, num_shards,
+ element_count_capacity, cache_value_time_extractor,
+ cache_value_check_timestamp, enable_prune) {
+ init_mem_tracker(lru_cache_type_string(lru_cache_type));
+ }
+
+ ~LRUCachePolicyTrackingAllocator() override { reset_cache(); }
+
+ void init_mem_tracker(const std::string& type_name) {
Review Comment:
should be private `_init_mem_tracker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167565397
<details>
<summary>TPC-DS: <b>Total hot run time: 171870 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1bb4f14c1c87bb8972bf9565ec19c30323d92e2b, data reload: false
query1 936 384 379 379
query2 6474 2378 2228 2228
query3 6651 205 210 205
query4 19088 17523 17574 17523
query5 4142 475 460 460
query6 262 162 160 160
query7 4601 291 288 288
query8 315 301 307 301
query9 8580 2395 2375 2375
query10 591 287 275 275
query11 10657 10152 9937 9937
query12 132 89 85 85
query13 1639 355 361 355
query14 9067 6130 6304 6130
query15 231 198 191 191
query16 7328 272 272 272
query17 1401 578 527 527
query18 1950 278 283 278
query19 199 158 158 158
query20 89 86 85 85
query21 208 130 127 127
query22 4419 4394 3996 3996
query23 33667 32905 33095 32905
query24 12446 2813 2894 2813
query25 696 355 356 355
query26 1903 152 155 152
query27 3213 320 311 311
query28 7835 2031 2021 2021
query29 1330 626 615 615
query30 283 151 152 151
query31 950 729 771 729
query32 92 53 55 53
query33 787 295 275 275
query34 1009 476 456 456
query35 731 614 632 614
query36 1106 934 963 934
query37 283 68 70 68
query38 2911 2758 2788 2758
query39 858 802 796 796
query40 283 126 126 126
query41 55 52 56 52
query42 119 101 105 101
query43 575 521 526 521
query44 1214 730 742 730
query45 195 159 166 159
query46 1085 713 715 713
query47 1878 1798 1773 1773
query48 386 327 295 295
query49 1202 419 404 404
query50 769 372 382 372
query51 6942 6775 6700 6700
query52 99 91 95 91
query53 354 282 284 282
query54 949 434 433 433
query55 77 80 76 76
query56 280 254 268 254
query57 1159 1039 1039 1039
query58 252 273 241 241
query59 3489 3179 3008 3008
query60 298 272 280 272
query61 100 94 95 94
query62 658 447 460 447
query63 312 294 284 284
query64 9999 2275 1748 1748
query65 3174 3087 3113 3087
query66 1419 336 328 328
query67 15397 14981 14955 14955
query68 4528 533 539 533
query69 460 316 313 313
query70 1194 1142 1127 1127
query71 400 278 267 267
query72 7254 5410 5512 5410
query73 755 323 319 319
query74 5990 5590 5542 5542
query75 3408 2698 2689 2689
query76 2687 968 972 968
query77 468 309 304 304
query78 10694 9867 9833 9833
query79 2517 508 512 508
query80 1096 466 462 462
query81 579 227 224 224
query82 758 105 97 97
query83 238 169 168 168
query84 238 85 85 85
query85 1632 291 285 285
query86 497 317 306 306
query87 3274 3086 3090 3086
query88 4237 2362 2363 2362
query89 494 376 388 376
query90 1775 198 184 184
query91 126 100 101 100
query92 69 48 49 48
query93 1795 506 496 496
query94 1265 199 186 186
query95 404 300 314 300
query96 586 261 262 261
query97 3208 3063 3068 3063
query98 228 206 198 198
query99 1209 864 844 844
Total cold run time: 275842 ms
Total hot run time: 171870 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2185620264
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz merged PR #36235:
URL: https://github.com/apache/doris/pull/36235
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167447786
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2164481302
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167574678
<details>
<summary>ClickBench: <b>Total hot run time: 30.72 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1bb4f14c1c87bb8972bf9565ec19c30323d92e2b, data reload: false
query1 0.05 0.03 0.03
query2 0.08 0.03 0.04
query3 0.23 0.06 0.05
query4 1.67 0.08 0.10
query5 0.50 0.50 0.49
query6 1.11 0.72 0.73
query7 0.02 0.02 0.01
query8 0.05 0.05 0.04
query9 0.54 0.50 0.48
query10 0.54 0.55 0.55
query11 0.15 0.12 0.11
query12 0.16 0.13 0.12
query13 0.59 0.58 0.59
query14 0.81 0.79 0.77
query15 0.84 0.81 0.82
query16 0.38 0.36 0.36
query17 0.95 0.97 0.96
query18 0.23 0.23 0.26
query19 1.80 1.84 1.66
query20 0.01 0.01 0.01
query21 15.44 0.63 0.64
query22 5.08 5.94 2.37
query23 18.26 1.29 1.17
query24 2.09 0.21 0.21
query25 0.15 0.08 0.08
query26 0.26 0.17 0.18
query27 0.08 0.09 0.07
query28 13.27 1.04 0.99
query29 12.62 3.27 3.23
query30 0.26 0.06 0.06
query31 2.88 0.39 0.38
query32 3.26 0.47 0.47
query33 2.86 2.96 2.89
query34 17.18 4.42 4.50
query35 4.47 4.46 4.44
query36 0.65 0.47 0.46
query37 0.18 0.15 0.15
query38 0.16 0.14 0.14
query39 0.04 0.03 0.03
query40 0.18 0.15 0.15
query41 0.09 0.04 0.05
query42 0.07 0.05 0.04
query43 0.04 0.04 0.04
Total cold run time: 110.28 s
Total hot run time: 30.72 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167435997
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2185620296
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2164457648
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "xinyiZzz (via GitHub)" <gi...@apache.org>.
xinyiZzz commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2164460176
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167566527
TeamCity be ut coverage result:
Function Coverage: 36.44% (8985/24654)
Line Coverage: 28.03% (73679/262902)
Region Coverage: 27.50% (38276/139182)
Branch Coverage: 24.20% (19521/80650)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1bb4f14c1c87bb8972bf9565ec19c30323d92e2b_1bb4f14c1c87bb8972bf9565ec19c30323d92e2b/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](memory) Refactor LRU cache policy memory tracking [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #36235:
URL: https://github.com/apache/doris/pull/36235#issuecomment-2167546132
<details>
<summary>TPC-H: <b>Total hot run time: 39817 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1bb4f14c1c87bb8972bf9565ec19c30323d92e2b, data reload: false
------ Round 1 ----------------------------------
q1 17618 4491 4317 4317
q2 2035 195 192 192
q3 10466 1122 1229 1122
q4 10186 860 693 693
q5 7475 2695 2639 2639
q6 221 137 134 134
q7 963 607 591 591
q8 9219 2094 2139 2094
q9 9203 6529 6531 6529
q10 8985 3738 3692 3692
q11 459 236 240 236
q12 519 225 227 225
q13 18930 2972 2939 2939
q14 270 229 218 218
q15 520 495 494 494
q16 516 377 387 377
q17 998 654 667 654
q18 8081 7424 7355 7355
q19 1699 1406 1485 1406
q20 642 309 319 309
q21 5039 3265 3946 3265
q22 383 336 337 336
Total cold run time: 114427 ms
Total hot run time: 39817 ms
----- Round 2, with runtime_filter_mode=off -----
q1 4302 4283 4257 4257
q2 367 262 259 259
q3 2968 2824 2742 2742
q4 1895 1604 1617 1604
q5 5258 5295 5287 5287
q6 210 126 127 126
q7 2122 1759 1777 1759
q8 3219 3369 3348 3348
q9 8446 8436 8419 8419
q10 3907 3623 3696 3623
q11 591 508 500 500
q12 786 608 595 595
q13 17447 2968 2997 2968
q14 294 278 252 252
q15 537 474 484 474
q16 462 400 409 400
q17 1774 1506 1490 1490
q18 7566 7625 7387 7387
q19 1717 1674 1585 1585
q20 1997 1787 1776 1776
q21 4830 4795 4726 4726
q22 613 580 534 534
Total cold run time: 71308 ms
Total hot run time: 54111 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org