You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/10/30 03:39:43 UTC
[GitHub] [doris] dataroaring opened a new pull request, #13781: [improvement](memory) simplify memory config related to tcmalloc
dataroaring opened a new pull request, #13781:
URL: https://github.com/apache/doris/pull/13781
There are several configs related to tcmalloc, users do know how to config them. Actually users just want two modes, performance or compact, in performance mode, users want doris run query and load quickly while in compact mode, users want doris run with less memory usage.
If we want to config tcmalloc individually, we can use env variables which are supported by tcmalloc.
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13781:
URL: https://github.com/apache/doris/pull/13781#issuecomment-1298470624
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #13781:
URL: https://github.com/apache/doris/pull/13781#discussion_r1009136506
##########
be/src/common/daemon.cpp:
##########
@@ -71,6 +71,14 @@ void Daemon::tcmalloc_gc_thread() {
// TODO All cache GC wish to be supported
#if !defined(ADDRESS_SANITIZER) && !defined(LEAK_SANITIZER) && !defined(THREAD_SANITIZER) && \
!defined(USE_JEMALLOC)
+
+ size_t tc_use_memory_min = config::mem_limit;
+ if (config::memory_mode == std::string("performance")) {
+ tc_use_memory_min = config::mem_limit * 0.9;
Review Comment:
I think `mem_limit * 0.9` is too big
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #13781:
URL: https://github.com/apache/doris/pull/13781#issuecomment-1296326999
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 39 seconds
load time: 555 seconds
storage size: 17154699352 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221030185614_clickbench_pr_35882.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #13781:
URL: https://github.com/apache/doris/pull/13781#discussion_r1009135808
##########
be/src/service/doris_main.cpp:
##########
@@ -321,22 +321,6 @@ int main(int argc, char** argv) {
return -1;
}
-#if !defined(__SANITIZE_ADDRESS__) && !defined(ADDRESS_SANITIZER) && !defined(LEAK_SANITIZER) && \
- !defined(THREAD_SANITIZER) && !defined(USE_JEMALLOC)
- // Aggressive decommit is required so that unused pages in the TCMalloc page heap are
- // not backed by physical pages and do not contribute towards memory consumption.
- if (doris::config::tc_enable_aggressive_memory_decommit) {
- MallocExtension::instance()->SetNumericProperty("tcmalloc.aggressive_memory_decommit", 1);
- }
- // Change the total TCMalloc thread cache size if necessary.
- if (!MallocExtension::instance()->SetNumericProperty(
- "tcmalloc.max_total_thread_cache_bytes",
- doris::config::tc_max_total_thread_cache_bytes)) {
- fprintf(stderr, "Failed to change TCMalloc total thread cache size.\n");
Review Comment:
What is the reason for removing `tcmalloc.aggressive_memory_decommit` and `tcmalloc.max_total_thread_cache_bytes`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #13781:
URL: https://github.com/apache/doris/pull/13781#discussion_r1009136506
##########
be/src/common/daemon.cpp:
##########
@@ -71,6 +71,14 @@ void Daemon::tcmalloc_gc_thread() {
// TODO All cache GC wish to be supported
#if !defined(ADDRESS_SANITIZER) && !defined(LEAK_SANITIZER) && !defined(THREAD_SANITIZER) && \
!defined(USE_JEMALLOC)
+
+ size_t tc_use_memory_min = config::mem_limit;
+ if (config::memory_mode == std::string("performance")) {
+ tc_use_memory_min = config::mem_limit * 0.9;
Review Comment:
I think `mem_limit * 0.9` is too large value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13781:
URL: https://github.com/apache/doris/pull/13781#issuecomment-1298470669
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz merged pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
xinyiZzz merged PR #13781:
URL: https://github.com/apache/doris/pull/13781
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] xinyiZzz commented on a diff in pull request #13781: [improvement](memory) simplify memory config related to tcmalloc
Posted by GitBox <gi...@apache.org>.
xinyiZzz commented on code in PR #13781:
URL: https://github.com/apache/doris/pull/13781#discussion_r1009134578
##########
docs/zh-CN/docs/admin-manual/config/be-config.md:
##########
@@ -839,6 +839,12 @@ txn 管理器中每个 txn_partition_map 的最大 txns 数，这是一种自我
* 描述：限制BE进程使用服务器最大内存百分比。用于防止BE内存挤占太多的机器内存，该参数必须大于0，当百分大于100%之后，该值会默认为100%。
* 默认值：80%
+### `memory_mode`
+
+* 类型：string
+* 描述：控制tcmalloc的回收，如果配置为performance，内存使用超过mem_limit的50%时，doris会释放tcmalloc cache中的内存，如果配置为compact，内存使用超过mem_limit的90%时，doris会释放tcmalloc cache中的内存。
Review Comment:
50%和90%写反了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org